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ABSTRACT \ ^ ■ - 

I This report describ^fi the pla^nnlcgi impleaentation^ 
and outcomes of 'the first phase of a..'^riDj€ct to create and laplemerit ' 
conputer^based laterials for a probleB^craented cuiiiCDlua in a 
training program for physician assistants. It exaali^s the underlying 
causes of the pro jectVs failure to achieve the Ph^€ I ctjectivesV 
and documents them as lessons learned and* situations to *aioid in " 
future projects* The information on which the* present rej crt is based 
was gathered inforrtailyi correspondencef documents produced by the 
Sheppard staffs notes made by corisultants during site visits^ and 
interviews with staff members are the iq j€|t iocrcei of data* 
Poilowing the introduct ion^ a brief history cf najor^events is given. 
The subsequent section identifies the fqur so^t slgnlflca|it autcoses ' 
or problem areasf and analyzes their causes* The final section' givf# 
a summary of the ^ajor outcomes^ briefly states seme additional 
outcomes^ and gives an overall assessment cf the Phase 1 effort, ' 
Appendices iliclude the request for trailing research', the 
Problem-Oriented Hedical, Curriculum (ECKC) i iplementation plan# the / 
POMC evaluation plan, and the proposal for"Phas€ II, 
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' ' ' ' t;/ INtRODUCTiON^ • A 

Background - ■ 

In November, t|73i a research agreement. with the title 
^ "Evaluation of PL IV /System in Air Force. Medical Training" 
was signed between 'representatives of the USAF^Sehool of.-Weal^t'h 
Care ScHehces ( SHCs'^ ab^heppard AFB,, the Air Force Human 
Res<xurces Labor ator^ ( AFHRL ), and the Defense Advanced Research 
Projects Agency (ARPA) of the Department of Defense. The 
agreement called for^i the research and devSopmpnt of a new 
problem-oriented medical curr iculum , ( POMC ) for the' Physician 
Assistant program offered at Sheppard»s SHCS, The new curric- 
ulum was to "^e developed and Implemented on the PLATO IV 
computer-based education system, A major^ objective of the 
project was to , be a comparative evaluation of the instruc- 
tional effectiveness of the new problem-oriented materials^ 
and the, existing traditional curriculum. 

Under the terms^of the agreement j the^Air Force's Air 
Training Command (ATC) was to furnish (through SHCS) a staff 
of subject matter specialists and support personnel to develop 
the new curriculum^ sufficient classroom and laboratory space 
to house the pro ject , and' enough physician aMlstant stiftents 
to form the evaluation . sample • " ' ^ 

AFHRL was to develop an evaluation plan in conjunction 
with SHCS and to provide direction for and monitoring of the * 
evaluation itself^ ^In additlohj AFHRL was to derive and 
implement measures' of student- and Instrugltor attitudes as 
well as tests of ^tud'ent pe^'formance * 

, It was to be ARPA 's responsibility to provide 20 PLATO 
IV terminals ^lus support costs for computer usage^ communi- 
cations lines, and terminal maintenance for ^ three years of ' 
service at= the Sheppard site. As well as providing hardware 
supports ARPA was to fund the Military Training Center s group ' 
(MTC) of the UjniVersity of Illinois' Computer-based Education 
'"Research Laboratory (CERL) to'^provide SHCS staff with train- 



ing and advice in the use of the PLATO system. The MTC group 
was also to coop%rate In^ the developmant of curriculum mater- 
ials for the project ;and perform other ^sistance as agreed 
to by the Sheppard project manager. 

In mid-1975i at the ;tim-e the Sheppard project had reached 
it^ .implementation stage , the project sponsor (ARPA) asked 
CERL to prepare a proposal tpido an evaluation of the Shepf 
pard effort as well as those of other ARP A-suppor ted PLATO 

'projec'ts* The MTC group had taken an important role in sup- 
port of each of the major ARPA sites. This role, included . 
giving initial training in the TUTOR language, acting as 
liaison with th^e rest of CERL, making site Visits^^for in- - - 
service training and problem solving, rev4ewing lessons to 
improve their pedagogical approach, etc. Since they had 
observed' these projects from their Inceptionj the MTC group 
was 'thought by ARPA to be in a good position to assist- in 
formative evaluation for the purpose of maximizing the effec- 
tiveness of each site, CERL's PLATO EducationaL^Evaluation^ 
and Research group (PEER) was also enlisted to .participate ''in 
the evaluation, ; ^ 

Another goal o,f the MTC/PEER evaluation which resulted 
from the contract with ARPA was to identify gener alizable 
aspec"t| of the experience of the A'RPA-supported sites. The^ 
evaluation sought to address the overall question of "under 
what conditions can CRE make a cost effective contribution 
to tne^ instructional needs of the services".^' PLATO implement a 

"^tiortT^t Chanute AFB and Aberdeen Proviag Ground sites were 
studied) along witH that at the Sheppard AFB site. Reports by 
Himwic/^ ( 1 9771^) , Call-Himwich .( 1 977b) , and Avner & Mi^selt 
( 1977 ) focus' o^ the experience at Aberdeen Proving Ground,* 
The Chanute project is discussed In reportS by [ftecka (1977a: 
1977br 1978), Himwich ( 1977a), Misselt, Himwich, Francis,, 
Avrier,' Tatsuoka % K*leci<.a ( 1 977 ) , Ta tsuoka , Misselt & Maritz 
(1978), and Tatsuoka (1978)- The report by Call-Himwich 
(1978) on the POMC courseware is a companion to this account 

n 



of the Sheppard project. , Other reports dealing with more 
general topics include, a , study of critical decisions mabe 
during various PLATO devtlopment projects (Steinberg etUl., 
1977)" and a set of? guldelinfes for site implenentat ion and - 
management ( Franc is , 1 977 ). , 

Scope and Rat-lonalf ■ ^ - . I \ 

The Sheppard AFB CBE project had two dis^nct phasesi, 
each characterized by its own goals, timelines, and organiza- 
tional structures. The first phase sought to create and impl 
ment CBF materials for a problem-or iented curr iculum in the 
sues training program for physician assistants. It. lasted 
from November, 1973, when the original research agreement 
was approved, until December, 1975 ..'when the staff terminated 
the effort due to , attrition in the student sample and other ^ 
problems which threatened the integrity of the evaluation^. 
After abandoning the first set of goals, the ' staff proposed a 
follow-on Investigation which was to^ involve preparation and 
testing of CBE niaterials for the Medical Lab, Radiology, and 
Dental Assistant courses in the SHCS, This second proposal 
was approved in time to begin development of^mater ials in - 
March, 1976. Final ^data collection f^r the segond phase 
ended on June 30, 1977, * . 

^ This report deals exclusively with the plans, transac- . 
tions, and outcomes of the first phase ^ of the SHCS proj^pt. 
By-v;rtue of the fact that the original' goals^ of the Vrojeq^t 
were abandoned (signifying^ the end of a first phase and' the 
necessity of a second), the project failed ' to ^achlev^ its 
Phase I objectives. The main purpose of this repprt is to 
examine the underlying causes of that failure ^and to document 
them as lessons learned and aituatiohs to avoid in future 
projects. It would, of course, be desirable also to examine 
the major events and findings of the second cphase , Unfortun- 
ately, the l^te completion date and CERL's reduced level of 



contact with Sheppard during 1976-1977 make it impossible to 
extend the scope of our evaluation beyond' phase I, A f ull _ 
repott of the PhaW| II effort, is ^to be matie by AFHRL^after" 
data analysis and interpretation are completedf ■ ' 

The information on which the present report_ is based "was 
gathered informally throughout CERL ' s' association with, the , 
Sheppard project, Corr esp'ondence , documents produced by the ■ 
Sheppard staffs notes made by MTC consultants during site 
visits, and. interviews with staff members are the major sources 
of data we have had available. The report fty project staff 
and AFHRL researchers ( Steinkerohner , Deignan, Waters & 
DeLeo, 1Q77) has also provided useful insight^ and we are 
grateful to AFHRL for giving us access to a prepubl ic^t ion 
copy. Although we have utilized numerous sources of informa- 
tion to prepare this document , we recognize that other forms 
of data (e/R,, interviews with course instr uctors , and closer ^ 
observation of implemf ntjt ion oohditions) would have signifi- 
cantly enhanced our under stand i"ng of Phase I, 

Much can be learned from Sheppard's Phap^I effort in 
spite of its failure to fully implement and evaluate' the 
problem-oriented approach to medical education,^ In order to 
focus on the ma jor . outcomes of Phase I, some -topics ( e , p, , , ^ 
the management of the courseware development and implementa- 
tion activities and. thei, ways in which CERL contributed to them) 
are diseussed only brieTly.^ However , much of this information 
is reoorted' el se where* ( see Steinberg et f 1 . , 1977;: Francis, 
1^77; and Steinkerchner et al , , 197^7). / : 

The report is organized in four major sections. Follow- 
tng the introduction J a brief history of major 'events is ^^given. 
The subsequent section identifies the four most significant 
outcomes or problem, areas and analyzes £hel^ causes. The 
final section gives a summary of the major outcomes^ briefly 
states some additional outcomes ^ and gives an overall 
assessmept of the Phase I effort, . ;\ 



- -^^^^ HISTORICAL SUMMARY ' ' 

Project Origins ' • \ ' ' ' • ' ' . ^ ' • 

J The Request for' Trainlrtr-Research (RTR) d^ted H April 
'I973. is the earliest official docum'e'nt generated during the 
Sheppard project (see Appendix A ) . - Prepared by an of f iclal 
In the ''School of Health Care Sciences". ( hereafter known as ' 
the "projeat"^^n4tlator") , the RTR outlined a rationale for 
the research efforr-and set out general plans for what was ■ 
to be attempted. ' -..^ . ' ■ " 

In essence, the RTR described a shortage of physicians' 
services in thp IJ . S. Air Force which could po^,ent,ial ly be ; 
alleviated through the training of large numbers of physicl'an 
asststants (PAsl. A training proRram for PAs had been estab- 
li.lhed at the SHCS in about 1970 with a comprehensive Plan of 
."Instruction being completed In August, 1972. In . the' view of 
the project initiator, the existing PA curriculum suffered 
from' the lack of a basic frame of reference for- the core of 
medical knowledge to be memor ized . , He proposed a currlcular' 
reorganization intended to illustrate the clinical relevance 
of this material . Although it 'is not stated explicitly, the 
reader of the RTR may infer that, a-main benefit expected from 
increased clinical .relevance was a decrease in training time. 
Part of. the impetus^ for a reworking of the PA curriculum was 
the frequency o f 'student quest ions about why a PA would need 
^to know certain^ basic science material. Hence ^ making the 
curriculum" rnore^tlinically re^:evant was ©xpected to improve ' 
student attitudes. Anpther supposed benefit was^that other 
Vllied health personnel wouldi^ain a better understanding of 
iheir own place along ^ the "di^nosis-treatment" loop, 

Hopes for redeveloping the PA qourse as problem- 
oriented medical curriculum actually preceded the plans to " 
use the PLATO system In this effort. It Is .our understanding 
that oTficlals In the, SHCS learned inearly 1973- that a set 
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Of 20'RLiTO IV terminals origin'ally intenc3ed"by ARPA for one 
of the othel? services ^mlght be available "for Air Eorce use. 
The projeet ^iti^torysaw this^as an opportunjity to fund the 
initial, work om^the POMC: he apparently was more .interested 
i n ^^vel opyife ' t he new eur r ic ul^um than in doing research with 
computer based educa t ron-- p^r se. The initial plans did, 
hov/ever , leav=e open the possibility that "non«POMC" CRE 
research ml^,ht be attempted, a^ is" Indicated by the following 
statement from the RTR: ^ . 

** . * 

Durinf^ the development of* this prd;]e^t, sttidents 
wi^dl be ex poised to bot h ' cur r ic ul ar f orniat s K POMC ' 
.1 ^anvi trad i tlona] ) in which PLATO IV may .6r may not 
be enploy%d. The ^u^i t ab i 1 i t y and contribution of 
cpmputer-assi sted nYstructipn will^^be evaluated as \ - 
^it stands alone and vis^a^vis th^curr icula , ^ ^.^-^ 

Durlnf?. the remainder of" the sprinp Ihd summer of K^73 
there were a number of meetinp/s between representatives of 
the .sues staff and tho^e of othe> .3fT,encies concerned with 
th'^ ^proJect ICFRI/, MC, and APHRD^in orddr further to refine 
ttie plans,. Becauser of the j^elative newness of the PLATO IM'. 
system and perier a^j lack of experience with large CBE curric^ 
ulum development proJectR, ARPA af^.reed to fund CFRL ^to provide 
support personnei for the .^heppard effort. Another event 
ciurinK thl^ perin^d was the attefirp.t. by the project iri'itiRton 
and (rther^ ^IDs on the PA cour^se staff to dr^w up n plan to 
ori^nni^e the PA curr'icnlum around P^i set5^ of syrnptoms ^or^ 
" prob 1 emf^*^' . Appnrontly it was still liopod that n full t^eor-^ 
Ranizati'iii of tlie qurrinulum coul d be/done , with C-Hp to he 
applied o'^rH y to n part tM' it. ■ . 

/ = 

Pr o j ec t^ OrMUui i-za t i on 

My Movornbor^j M^Y s , n tVHMDnl resonr'oh rif^reompnt had been 
nip;nnd and thn projoot wan offlrinlly undnrway. It wan also 



,'iboiit till.:; t iiiiij-- ill ,1 1, tht- pi'DjinM i ti i f, I .1 1 or- f i 11 ;i 1 i /. (h1 
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to leave the Air Force and an experienced adm'in ist rator was 
teelQcted to become acting director. In Daceinber, 1973, the,= 
acti^ director, the project initiator, and two other SHCS 
officials participated in planning discussions with memb^ers 
of CERL»s MTC support group. 

. During the remainder of 1973 and the first f^w. weeks of _. 
^1974, a number of organizational and staff ing decisions ^ere 
made. The acting director , hav ing taken up residency at 
Sheppard AFB during this period, arranged for pffice space to 
h^use the project and its PLATO IV t erm in^al s / He also selected 
a num^ber of staff members, some of whom were to join the 
project in February with ot^hers not coming until June or 
July, , ^OJn^ 

A aeries of planning, meetings was held at the SHCS in 
late February. Representatives of the MTC sroup met together 
with personnel from Sheppard *s PLATO IV staff antf instructors 
from;.the Physician Assistant's course; Among the , major out = 
comes of these meetings l^ere: (a) a working definition of 
jC^rob le)n^ or i en ted med ic al curriculum, (b) an understanding 
that the PLATO project would constitute Hhe core oT the 
research on the PA cdUrse (and tha.t .it was beyond "the scope ' 
o'f th^ project to undertake the revision of the PA curriculum 

n whole), (c) an indi6atLon"from the project staff that 
the evaluntion of PLATO IV as an instructional medium would 
be secondary to the evaluation of the POMC,' (d) disclosure of 
plans to divide the development '.effort into a basic science 
conlponent and a problem-oriented component, (a) a 'description 
of the intent to use PLATO in a wide variety of specific prob^ 
lem areas within- the course, and ( f ) discussion of the role 
that might be taken by CFRL*s MTC group. • 

While awaiting the arrival of the pr imar y:. sub J oc t matter 
j^pec in 1 1 Hl^, ( n no wl yj^^fu^ad uated physician),^ the staff members 
nlr^eady "ori boa r ci tr a v u 1 ed to CK^l to receTve trnlnlng in 
t\\o TW'Vnn Laruuingo:, i n s£ rMjc t i o rm 1 design i hnd other n5^pec^i> 

I 'I 
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of the use of PLrfTO IV (see Francis/ 1976 



return to 



Sheppard AFB^ they con'tinued to develop their skills' in le#sbn 



writhing wh i 1 e ,ref i n 1 n g the plans for the nev/ curriculum* 
Work on a number of basic science lessons was bef|un while 
awaiting the physician'^ the clinical" subject matter expert 

The weeks following the arrival .of the physician were 
marked by a significant alteration of the project's direc- 
tion. Whereas it had previously been planned to develop CBE 
materials for ea.ch of a number of topics which were thou^?, ht/ 
to bfe* well^surted for CRF application, it Has decided during 
this period thiat all of the material relate oT. o a s i n f, 1 e 
orr;an system (the respiratory system)^ should be targeted for 
, t h e PO M c , Th e po r t i o n s of the c ours e P 1 a n o f Ins t r u c 1 1 o n 
( P 0 I ) dealing w i^ t h r e s p i r a t i o n were identified an d these 
t o n i c s were re o r p, an i z e d ^vn d reordered f o r ' '\ i n c o r po ration in t o 
t h e ^ 0 M.C . r o n c u r r e n t wit h t h e c h o'^'i^^ o respiration as s u b - 
J e c t , matte r w a s n re c o n s i d e r a t i.o n / r e d e f t n i t i o n of the P 0 m c * s 
'nature. The subject matter specialist was given primary 
r e 3 p o n s i b i 1 i t y fo r p 1 a r\f\ i n r t h ^ c u r r 1 c u 1 um , and he r e i n t e r ^ 
preteii the d^^'finttion in a manner with which he could feel 
f"^ o m f o r t able, A f u .1 1 e r d i s c i \ s s i o n of this po i n t is i v e n 
1 a t ^ r 1 t h e Q fi o r t . 

Initial develo'hment work on elements of both the basic 
S(Menre componpnt and a clinically- oriented portion of the 
POMC ■ |) ro'^ oed cmI thrM)u^h tfie !Ujm:ner^ and fair of JQ7^i. ' hmrww 
thp major^ event^^ of this period were: (a) the arrival o{ 
t h p r' e m a 1 n 1 n f', t a f f !ii e rn b e r s :^ 1 1 o r a t ^ d t o t li e r o ion t ( t h e 
1 ckU. of whom was not as^iigned until Nov cnber^-one year aftrr 
the project pot urfderway),, and^(b) the initial platinin^^ for 



the" implementation and evaluation [^ha^w' 



Plrinniri'^^ f^c^r- the [m|f\ omen tat v^Hi - 

Slnnew^lans for r ed e v eTap i n tfie entire f-A eurricurnm 

to be pr'ob 1 em-or i en t.ed jiad tieeti abandoned or postponed, th( 

L Ml em eri I a t t o !i ef J h*^ POMf" lind to be done in :^unh n way as 



1 ;j 



) which calied fo|r the 

with 



nent a- 



to minimize disruption of the regular courge% . An ,imple 
tion plan was drawn up (see Appendix 

replacement of d i sc r'e te hour s of "the regular curriculum 
new CBE rfiaterials. In order to provide^pr a comparatilvf 
evaluation, the pro Ject ^ staff planned t^ select a sample. of 
16 students for the experimental (pdMC) group' with the/retnain» 
ing U8 students assigned to the regular curriculum. Only 16' 
subJects^ were to be assigned, to the %x per imental condi/tion 
because there were only to be 16 PLATO IV terminals for stu- 
.dent use, and it was believed to be necessary to have/a" 
student to terminal ratio of 1 to 1 . To increase the' size of 
the n, however/ the implementation plan provided for a 
repliGatiQ.n of ttre conipar i son with the kjsG of a second class 
of PA students. Hence, a total of 32 stiidents were to be 
exposed to the POMp^ dur ing\ the evaluation phase. An impgrtant 
consequence . of the decision to use a second class of PA train- 
ees w^as that the start of the Lmplementat ion Tiad to- be^ moved 
forward from Jajiuary,^ 1976, to October, 1 975, and' finally to 
July, 1 975. Hence, the tirTLe^^available for lesson development 
was shortened by six months. 

An evaluation plan was prepared in March, 1975, to out= 
line specifications for data cpllection and analysis durine^ 
the comparison (see Appendix C), It called for the use of a 
comprehensive examination over the resplA^tpry topics covered 

The. two groqps were also to be compared 



in both curricula 
on their performance in a set of simulated patient' encount^^rs . 
These latter exercises were expected to measure th^' problem^- ' 
solvinf^, abilities of the PA trainees in the task of dj^iw^nosis 
and treatment of respiratory diseases and other patient ail=^ 
mejits; The ma^asurement of student attitudes toward /the POMC 
and toward q o ni p u t e r ^ b a s^e d '"e'p'uq t i o n w a s a 1 s o a n i m^r t a n t 
component of the evaluation. ^The plan cilled for two adrninl^-- 
trattons of on attitude questionnair6'--one after .a brnef ^ 
exposure to the POMC and the other after comp 1 ^t i on . o f the^ 
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first year ot the PA^ prog nam . /The rel i^bil ity^^^nd incorpor- 

also '^o /be investigjated 



ability of the PLATO^ system 



Resul ts of the Impl emen tatl/n / ' J 

In spite^ of the fact /hat some/of the CBF les^.ons slated 
for first trimester use w/re not ypt ready, the implementation 
got underway in July% ^ 9/5 t as scH'eduled. The first ^lass's^ 
initiaf use of the C9E /essons did not smoothly. Students^^ 
encoifntered a numller of obding arrors in the' lesson^g^,- and 
there was an unfortunate mixup/amon% the project sfaff 'as to 
the policie-s on student dre^s/a^!d behavior in'^tHe PLATO^labora 
tory,, Thf^ atudents/wer^'- f€ r^t/ told by some staff members 
thai an in formal ^a/mosphere be maintained. This ^'policy 

was quickly r esc i /ded , by . tW^ project director , who felt that 
-frequent visits from Air Force ^^^^brass" would require conven'*,. 
tional dress ancf attendance patrtxrns 

Schedul Inf Qf. PLATO ly ^udy sessions was a ditficult 
problem. The pb students irf the experimental' grou^ were -"A^tJ'^ 
'excused from £he regular class sessions in which respiratory 
topics were^io be discuss^ed in order that they might; receive 
alternative^Unstr uction on the PLATO IV. system. Since it was 
J;_necessary fjr them to rejoin the regular class at the next 
(nonrespiratory) sessio'ns, they scMetimes were not able to " 
complete |he assigned CRF: lesso^^^^in the available time.| To 
make up for. such lost study time and to ^have opportunity for- 
review Ojf 'CHF: materials, th'e- POMC s^tudents often had to return 
bo the t^I.ATO IV classroom during eveninfT, hours/ 

Thie J ate summer months of 1Q75 were ver^busy ones for 
the prc|ject personnel. The severe time/ pressure on the ataff 
to comlienCe student use in Ju 1 y^J. wh i c had left too little ^ 
time ipr d ebufTg i n g" of programs anil . coord in ation of policies 
r^garc|ing' student dress and behnv iorT ' con t inued th-rough the 
suntmeif and faM. Not only did the st a f fr ha v e ■ to rus4i tJ^ \^ 
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lete the fa r^t trimester lessons Vov the initial group 



of stuftents, they also had to prepare for* the' start of the 
secohd trimester/ A aumber o/f second trimester lessons 
remained to be ,compl e ted.^ and the first tr imestfer/ les^sohs- 
required revisions prior to /being used with the nex en ter ing . 
class. It was onl y t hrouf^hj go nsid er abl e effort and much 
qvertime work that the project staff was able to meet its 
.schedule,. Nevertheless, the second^ trimester goi underway 
with a second group of 16 studerfts being assigned to the POMC 
condition. By this time there ha-d been s^me attrition in the 
first class's experimental group so that only eight students 
continued to tha second trirnester's material., 

Of the eight students from the first class who had left 
the POMC Rroi>p.by the st^art of the second trimester, five had 
b,mh^ Bcad-emically el im inMt ed f rom the PA program. Three/ others 
'had remaih/ed^in the Drogram hut had requested to be dropped 
from the .Experiment in order to r,eturn to the regular class- 
room. Before the second trimester was compJetedj six of the ■ 
remal/nin^ ei^ht ^fe^ent^ al so asked tp go bfeck ^^tb /the cl ass» 
room. As with -the' first class the second set of PA students 
reacted negptively to the ex per imen t a 1^^ c6 nd i t io ns'. Realiz^^ 
ing that "nothing further could be done to 'resolve^ the miscon» 
ceptions, relieve the anxieties^ or change the negative 
attitudes of the students", the staff decide^d it would be a 
waste of time and money to continue the project wi%h so few, 
.students and voted to term^inate the POMC effort in December^ 
^19^5 (Steinkerchner it' al . , 1977). 

At the time of the decision to d i scon t in ue 'the P0>1C , 
most of the CBP le^|*ons planned for. the first and second 
trimesters had been completed. ^ Man^y of the third trimester ^ 
materials, however, had not. Some of these^ latter Wessons 
w%re to- t%^ke^.,t,he form of simu^ate^f patient encounters.^ Si\nce 
tbe basic format br the V^-simul at ion s had been determined 

earlier, ail that remained was to input the "data" from a set 

* / ■ ■ 

of actbal clinical eases* The. remaining third ^ tr imester ' " 
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leFp5ons,= on the d^her hand, would hay'e requ^irad -more effort 
to develops It i^. possible, biVt noy' likely, tha^ they would 
have he|n cdmple|ed in time Tor use/had jthe POMC students -not 
opted to leave t^ne experiment. / y ^ 

Because of/the early termination , much of the evaluation 
data called fory bv the evaluatior/ plan (Appendix C) were not- 
colLppted. ThprR was no d i r ec t /com pa r i son-^ f the POMC and • 
regular curricjula with re.^ard to instructional ef^f ec t i veness ^ 
Neither the kpow]edq.e of h a s i c ' sc i enc e ^ ma ter i a 1 nor clinical 
p r ob 1 eni- sol V i'ns, . sk i 1 1 s vver e ^comp^ared between the" two groups. 
The at t i t ud i n a 1 " vda ta were^ collectpdj but the second adminis- 
tration of the q ' i e s t i o nn a i r e c ame at the time of the dis'con-- 
tinuafice of the POMC^rathar than at the end of the third tri- 
mr^st^r ns ha;i oriRinally been pTanned, (Jn fo r tgn at el y , tKe 
basic que^tidns rcfiardinf^ the value of a pr ob 1 em^o r i e n t ed 
curvrlouluim in ph ys ic i'^n as%i Itant training wer^ not answprpd 
^ '5 V t ^ 1 e d n t a c o I 1 e c t e d d u r i , n the h e p n a r d p r o j e c t , 



ppr. r^n'iriirip for Phase 11 

At ^bout the time that the Sheppard staff decided to 
cUise^oiit Rx per impn tat ion with the POMC, ^ARaPA and f^RP c^me 
to ary af^r cement that the MTC :^ rounds mjnpor t ,rol e .would be 
"Mjrt/jil^d. Althou,f?,h. th(^ '^TC p^ro\}p, had little opportu^nitv 
liri'i^r 1 1 nhanned rol^e to ke*^p abreast of the plannini^ and ^ 
imp4^^mpn.,ta tMOn n.f nhe.ppard^s n^cond phase r?ffort, it is 
i fKp o r t a n t t o-. indi n F\t4 snmethinf"^ of the naturp oT thf 
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a n n i n 



in "[^# c e mbe r 



^ Q r t h e f o 1 1 0 w.^ o n p r o j o n t ( o r 



s e c 0 n ( 



p h a : 



n e r a n 



.'L o Q k i n p f o r a a i ) r s p r\ in w h i a h t h ere w a s a 
^ if^jQU .;'t i.id.etn.t _f^Jnw a/id in- whirt a hlph level of faculty involve- 
ment.war likelv, t.\'j.e nheppard staff selected the Medical Pab^ 
■' R^n '1 i o 1 o r; V , a n f e ri t ri 1 Assistant: courses within 'the S f i C S . ■ 
^H. h^r pl:"j^nnlnt^ cn fvs i d a 1 1 o n s wer'^: fa) the renuirement that 
identical cri,terifpn mensuren bP administered tb, CHP and^con-^ 
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;o.n test and' block test levels, (h) 



h^oi'outr[i r^e a sy r c/ile n t o f ^ 1 e ar: n e Si'v eTfCpT^i n char' nn te r i st i e 
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r and (c) the capabiiity of running \separate CBF and ' control 

^?,^oups far^ an entire bloGk of instruction CSteinkerc hner et 
al., 1977). ^he criterion of high instructor involvement 
.demonstrates that the staff wanted to identify courses whose 
instructors felt they had problems that could be'solved by' . 
use of PLATO. The PLATO staff , had lenrned the importance ^ of 
high instructor involvement from "their experience in the POMC 
phase of the project. ^ 

Development v/ork on the new materials bega'n in March, 
1 976\ with preliminary data collection scheduled to take 
place, later that summer. Pecause the ariginal contract with 
ARPA was to expire September 30, 1976, an extensio^n was funded 
until June 30, 1977/ Formal data collection for i summative 
.evaluation was to commence in November, l'976, and continue 
until the ead of the ARPA-funded extension. A final report 
v/as soheduled for cdmpletion by December 1977. 

According to the research agreement for the extension 
period tsee Appendix D), the main goals oKthe follow-on 
. e.ffort and its evaluation were: (a) comparison of Instruc- ^ 
tional effectiveness of CRF:- lessons and lessons, presented 
by lecture or by^ conventional self-paced media, (b) determi^a- 
.tion of costs of DRf; and non^CRE instruction in the .three 
cours^es,/ (c) determination of .factors underAyjJthg student 
V ^attitudes toward CBE, and (d) determ|nat iom of factors under- 
\ lyin"R attitudes of ihstr'uctors toward CBF. 
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ANALYSIS "of PRo/rCT ' OUTCOMES 



Student Withdrawal from Ex pe r iment 

__^= — ^ 

The major reason foV the staff^s decision \6 abandon ^the 

*POMC effort was the heavy loss of subjects from'the exp^ri^ . ; 

ment^ group due' to academitf failure and dissatisfaction with 

the experimental condition™ As stated above , fdve of the 

orifz,inal °1 5 POMC students OH) were elimihated from the PA 

program due to p^or academic ' per formanc.e . Only six of the 

control-group st ud ents ( 1 2 . ) were dropped during the ^^satne 

period, ^ In spite .of its higher drop^j^ rate, the PQMc|group^s 

five ellminees did not- feel' that , their failure was related to 

participation in the e^pariment. Instead, they attributed 

their problems to rapid pacing~^ln the regular portion of the 

course ( S to i n ker c hn er et al . , 1977). Nevertheless, the 

losses due to academic attritton^ dealt a severe blow to the 

-_ _ i> ^ 

projedt's chances .for a successful outcome. 

Al thoug.h ■ the academin el im 1 nj^.t i on s m^y not have been 
related to the POMC 'or its implerifiSn tat ion , eight oX the 
remai^ninp. eleven 'students in the f i r st.-»Gl aps aske d to leave 
the p8mC groij^p because they were unhapp^y wi<^ the experimental 
conditions. This mass dissatisfaction wa^ expressed' informally 
at first through individual complaints to the Shep^ard PLATO 
staff. Ijltimately, however, the ^Issident group submitted 
formal letters of request that they be transferred back to\ 4 
the regular classroorH. This latter action prompted the 
project staff to termWa te. the" pqi^^C ^ e f f or t . 

' The Mjndej't y ing cauf^es of student d i 5^ ^t i s f ac t Ton^ - pe r h a ps 
thr most strikin^^ outcome of the POMa e f for t»-are numerous, 
but they are rooted in the severe impl emen ta 1 1 on'* con st r ai n-t s 
UmpDsed on the project. A discussion of these constraints 
must begin with m understanding of the students* perspective.^ 

An Physininn Assistant trainees, ^he students were a ^ 
ht phly-mot ivategi nn^i s t r ong 1 y-^ tjoa 1- or i en ted group. They had 



( '15 , • 

■ I 

- I % 

all worked in health care specialities during th^ir Air Force 
or Navy careers and many w^re hopeful of working as PAs after 
leaving the ^ervice*^ Career aspirations ' and family prfesaur^es 
made success in the PA curriculum a very ^ important objective* 

Because nearly all^ of the academic training was covered 
in the first year of the tw\) year program, the general pace 
of the course was rapid and tightly scheduled , /^Students spent 
six to eight hours in classvt)rk each day and v/ere expe'Cted 
to co'mplete rea^in^ |iBSighments and outside study during the 
evening hours and on Weeken4s, Most participated-^ in sm'all 
study groups to reMifw lecture notes' and prepar'e for the , 
frequent examinations in the varipus subject areas. Hfence*, 
there was am pi e opport un i t y for' them to exchange YievJB on , 
ihstri3ctO'r 3 j ev,ents';0f the day, and -their prog.ress through 
the curriculum'j as well as to discuss spec i f isC it ems o^ 
counse content, . ^ > ' \ . = 

The POMC impl emen tat ion plan (see Aopendix 3) cplled =for 
the replacement of discrete hours of cl assroom lecture with 
newl'.y-dev e}.ope/d CB^ lessons. The 16 students selected for 
the POMC group (one fourth of the class of 64) were to leave 
the classroom/ during the times the target respiratory material 
were to be pr|esented in order to study newly-^created lessons 
from the POMOr^,^, Sinw the POMC involved a reorganization c^f 
the respiratory content of the curriculum, the topic studied 
by the experimental' groyp on a particular day was often quite 
different from that covered in the r^ular classroom. 

The PLATO staff tried to assure the experimental group 
students that although the PO'^^C would present topics in a 
different order, eventually all would be covered, Nevertheles 
the students were frequently disconcerted to find that the 
classroom irfstructor had presented "new" material that they 
had "missed" whil^e attending a PLATO session, ^Because of 
their compulsion to do well in the course, these students 
often felt that %iey had to work harder to "keW up" with the 
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f^las5^ This presunably meany that, they spent time studying 
the lecture notes of friends/ on in doing **extra*' reariinR' 
assignrTient s,- The ^Doncern asout missing lectures was so great 
that' some students considered the POMC to' be' an additional 
course rather than an altennative presentation of so^ie of the 
same ^ ma t er i al . 

We have not had access to the letter | of ^*mut iny" , but 
it is apparent from'the interview and questionnaire data « 
summ^arized by Steinkerchner et al . C 1 977 )- and the informal 
reports of project members that the major ■.complain t of the 
students was that they felt they were not ^.etting the same . 
/InforTiation that- presented to the rest of the class. The 
second most frequent compl aln t was d Irectpd at the test grad- 
in^ procedures ,used ^to accommodate the experimental conditions 
AccordinfT, to the nlanj the regular instructors^ were to meet 
with a PLATO staff representative t,o discuss each test prior 
to its adm in i s t r g t ion . If the test contained items that had 
not vet been "tau0,ht^ in the POMCj-'^the instructor was to identi- 
fy them As -items to be omitted bv the PO'^U: students. The 
PLATO ^staff'j on the other' hand, was to be responsible fpr 
testing the POMC students over those items at a later date. 
The test scores for the POMC proup were to he recorded as. the 
percentaR^ Of ■'non-omitted" items that were answered corrlectly 
Scores were later adjusted to re f 1 ee t per f orm anc e on the 
make^-up tests given by the PLATO staff, .The procedure was 
cojnplicaLed at best and it was not implemented smooth! y-^-at 
least 'one instructor in the course waS' never told how it was 
'to wo/i<. Stude.nts distrusted the process and, because of 
their g'reat concern with course Rradeslj they were often upset 
a*t not being r i y^n - cr ed i t for correct^-" responses to "omitted" 
items^i* 

' Another stuf^^ent complaimt was that they often lacked a 
sense of where thev were gci^ir^ in the POmc , i.e., the/ did 
not have a connept of the overall organ i zation of the new 
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curriculum, ^fhe uncertain rate of aeuelopment of future.! 
sons made it difficult.'to publi.^h a* comDrehpnsive sched.ule 



es 



or sy\j.abus for student use. Furthermore, the scattered 
d isti^ ihut ion of respiratory tqpibs';)in the regular Plan of 
Instruction, and hence the sporadic scheduling of PLATO *^ 
sessions, made the POMC appear esp^c i al 1 y disjointed and 
fragmented. Because of the lengthy intervals bftv/een PLATO 
sessions, students followinn a tichtl/-knit argument or pre- 
" sentntion in the POMC sequence often found it necessary to 
review previous lessons^ before continuinf^, through the curric- 
ulum. * ' 

As mentioned earlier, the>OMC students ultimately 
stated that contact with the PLATO staff was amonf? the most 
^positive aspects of their experience, but.they;may not have 
been favorably impressed at the oijtset/ They wene told in 
their^nitial orientation to the project that they would have 
certain freedoms in dress, attendanae patterns, and behavior 
while in ^the PLATO classroom. These raised expect^^tions 
were soon lowered attain when the project director' returned 
from an out-of-town assignment. Real izing tJiat'' the project 
would have, high visibility (particularly among high ran4<ing 
visitors to the SUCS) , he rescinded the promised freedoms and 
insisted instead on a high standard of military decorum; The 
impact of this^mixup in communication among the staff (i.e., 
fallinF to settle on a consistent policy for student drpss 
and attendance prior to the initial or ien t a t i on ) i s dif^l^ult 
to gauge. Doubtless, however, it did not leave a favo/able 
impression.' 

Another problem that dampened students* initial enthusi^ 
asm for the CBF: project was that the lessons had not been com^ 
pletely ''debugged'' . There were still a number of codinj^ 
errors which detracted from the instructional presentation. 
These errors and the policy mixup described above were prob- 
'ab^y-all due to the great time pressure placed on the project 
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staf^ by the implementa t-icKi'-'schediile they adopted. There wa^ 
too little opportunity to ^^rio adeei'uate testinr, of the lessons 
and to plan for a smooth i m p 1 em e n t a 1 1 o n . . ' ' ^ 

There wer'g also a number of complaints sbout lesson qual4 
l£v. In reviev/infi, the discussion by Steinkerchner *et ( 1 97\ 

one rri a y f o r iti the i m p r o s s 1 0(n that m a n y o 0 these go m p 1 a i^n t s 
were due to an 'abnormally high number of communicat ior)^^ 1 ine 
.errors (a troublesome but sf 1 v ab 1 e ha r d war o problem) rather 
t h a n the in s't r u c t i o n a 1 content o f t Ki e 1 e s s o-n s . Wot a 1 1 > o f 
the lesson quality complaints can be so easily d i s m i s s e 3 ^ 
however J because the qualitv control procedures were not rig-* 

0 r o u s a n d r e n o t s t e m a t i c a 1 1 y enforced. To a 1 a r e e x t e n t 
each author 'v^,as allm>;ed tp make his own Judgments as to *when 

1 e s s o n s w ere r e a d y f o r s t u d e n t u s^e i n the c u r r i c u 1 urn , m e 
lessons were tested on members of the ener^^^l tarftet ponula- 
tion, but nany ware not. Mothinr. like the *tvalddat ion prope- 
dure suRqested in the Air Force Instructional Systems Desapn 
m a n 1 1 a I s w a s a p p I led t o^ t h e S h e p o i r d 1 e s s n n s -. It mi r li t v/e 1 1 
be ex p e c* t e d , t h e r r,f o r e , t h a t h e r e v; on] d h e s o me p e d a r, o p, i c a 1 
dpficiencl^s In thpse basically "untested" m^teriHls. flever- 
t h e 1 e s- s , it is a p p a r e n t t h a t n e r c ^ i v e d s [i o r t c o m i n f s in 1 e s s o n 
q u a 1 it V we r e o n 1 y a s e c o n d a r y s o u r c e o f d i. s s a t i s f a c t i o n , The 
m n j o r p r o b 1 e m s s e e n by the s 1 1 1 d e n t s h a '\ t o d o wl t h t h e rf a t u r e 
of the exnpr/fm'^nt*s implemeritatian--not the quality of thie 
POi^^r lessons. 

Most nf— ^hesp student eompl^ints seem to stem from n 
feelinp of^bein'; ui a minority of "^Miinen pi^f^s" who were beint^ 
treated differently than their peers. Given the rencttve 
arrangements of the P'^MC Implementation, It is not difficult 
to u n d erst a ji d w h v t hey may h a v e h eld t h i s feel i n i\ . P e r h a n s 
their a n x i e t y wo u 1 rl h a v e b e e n 1 e s s n e.r v a s i v e h a d 1 1 b o e n "' 
possable to nssipn a Inr^i^er^ sample to the POMC condition. 
If half or ^fTrore of the students had been expensed to the CVW. 
mater'ials, it is likely that they wcMild have been less con- 
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cerned about missed class sessions, study of topics iha , 
.. different onder, and 4unG%rtainties ab^.t o-urricglar organiz-, 
ation^ BeinB in , the majorit-y, they may'no't hav# -felt anxiety, 
that the'y w^r.a being treated differentLy than (and perhaps . 
not as well' as) the stud-ents in the traditional ourr ioul urn . 

Why,, then, were only ong fourth of the s^dents assigned 
fto the POMC? Basically, it was because the high information 
content, and- rapid pacing of the vPA curriculum led to rigid 
scheduling ,of student time. The fixed schedule meant that 
there were relatively few free 'hours during which the students 
were able to use the PLATO terminals. To maximize terrrTtnal 
availability when they might he needed, the project staff 
• decided to assign only one student to the experimental group 
for each student terminal available. Four of the 20 te»minals 
were used for authors, leaving' only 16 for student use-. Hence, 
the experimental group n was set at 16. It should be noted 
that the rigid sche.duaing of student time--not the number of 
terminal3--was the, trtie limiting factor. Because the students' 
time was so structured in the PA curriculum, they couldn't 
make efficient use , of the . term inal hours that were available. 
Other approaches to the scheduling and Implementation problem 
were considered, but 4^ere Jiidged to be even more fraught with - 
difficulties than the implementation approach described here. 

-f" summary ,^ t^\e POMC phase was curtailed because of stu- 
dent dissatisfaclzion with its implementation. The implementa- 
tion placed a great strain on the experimental group stodants 
because they were a minority and were treated much differently 
than their peers in the regular curriculum. The reactive 
arrangements associated with the ■POMC "minority" were neces- 
sary because of the fundamental constraints on student time 
in the PA program. Thus it was these course constraints 
which were the basic cause" of"P&MC effort's downfall.- 




Inootnplete Devalopment of tHg^POMC ' ^ ^ , , 

.tven if thera^had been no "mutiny" as the first class 
of POM.C students epmpleted the second trimes^ter of their pro- 
.:{gram, a;t is not Wrtai/i,^^hat the .remain ing thJtr^d trime'ster 

les4olis<^ would have been ca,nipletGd . in time . The fact that 
= ^ iesspri/fjroduction r^tes fell behlna schedule must therefpre ^ 
B,e second ma'Jo^r pro-ject 'gutcome to be considered. 

The original research agreement ^istated thit 200 hours 
of C^E instruction would, be de\)teloped for tft.^^ problem=or iented 
curriculum. Later .at the time! it. 'was"" decided to focus on 
the topic \of respiration, the staff reduced thfe total to 1^5 
hours* ThiS; was felt, to be justified in that the 200 hour 
'figure had been arbiCi^ar^ly sfelec/ted ' and becauseVthe smaller 
ril'ure represented .approximately^lOfv of the' 14'00 hours com- 
prising the first year of the BA' program, - According, to the 
implertentation plan (jee Appendix D) , the T^'S^ Plgn of Instruc- 
,.tton hours were to include 121 hoUrs of clinical and* basic 
.science materials distributed throughout trie year (i.e. , 2^1, ■ 
25, and 72 hours across the three trimeste^Sj respectively), 
' In addition^ there were to,W^ 24 hours . of |Bptient encounter 
simulations whic h\ had no counterpart in th^= regular PA 
curriculum. / ' %: - 

By the'end of July, 1 975 , howf ver ^ there had betn ©' 
further reduction in the amount of m.aterials to be^ d ey^fe'Lop^ , 
A project document prepared fqr ARPA indicates that 104 hours 
of clinical and basic science lessons (22, 29, and 53 hours'* 
yTor' the Ist, 2nd, and 3rd trimesters, respectively) were 

planned. These, p^^us the 24 . hours of patient encounter simu- 
. lations, gave a projected, total of 128 POI Kqurs of materials.. 
yThe rfport prepared^ for ARPA al& '^gave' estimates of the 
pis^Ual clock hours to be consumed in studying the CBE materi- 
*als* Whereas the earlier estimates had^ been reported in 
/ten^ms of POI hours allotted in the course documents for the- 



respiratory topies,. the project staff belleved'tha^^ fewer * 
clock hours would^b« n^ftded for the CBE lessons* To arrive* 
at the estlmatf^ of/ qlook Tiq^urs , the sta^f .assurri^ed that. 251^ 
of the POI time could be saved becaus:e proRr*i^B through ^th6 
POMC would be'^^felf^paded^ rather "€jian'^ j^ albh 
assumed that an additional 1516 c^uld be savfd iWltji^GtiE as^ 
compared to- bther 'seliT^pace^ media. Without quesiioning the 
basis for these' assuinptlfan4 it Is interesting to rtote that 
the staff proJected'MI ^13.1, an* 24.2 clock hours of 
replacement POMC lessons to bm mbbO^ in the 'first, second, and 
third trim,esters^^ respective^* These values are considerably 
smaller than their Pof hourAounterparts given in thi same 
doGument-(i.e. ,^22/.;^2Q, and; 53) or in the implementation ' pi an 
itself (24, 25, and 72).v:^o separate clock hour estimates of 
the quantity of simulated patient encounte-rs was offered; the 
original estlmate^of 34 Pni hours was hot mod ified / 

' ^^.V^ile the reduction -in' scope of th% project and the prob- 
.lem of whether to describe it ^in terms of POI- hours or clock 
hours are interesting obse^A^at ions ; it Is more important to 
/examine the proportion of the total mater ials that were com- 
plettfd'. In Jul.yv 1975, the Sheppard staff reported ^hat ^ 
while BIS of the first tr imestdr ressons had ^been flnisheti, 
only 361 of the second trimester and 18^ of the third trim'es- ^ 
ter lessons were' in a usa^b^e' ;st ate . The remaining lessons 
apparehtly ^viere in various stages of completion at that time. 

No mention is made by Steifrfeerchner et al , ( 1 977 ) #-s to 
the proportion of thf, lessons that had been completed at the 
time the^^POMC was terminated. Since the first and second 
trimester materials were the first needed for student use, it 
may^ be assumed that ths^ development effort was concentrated 
on getting those lessons^^produced * If they were completed 
(and we don't know that they were), it would have left the . 
bulk of the third trimester lessons to be developed in a 
relatively short period of time. Had the POMC phase been 



'^(teontiniied , It ts conceivable but not likely that Ahe=:.fehird 
Vitrimester lessons could have •b#e'n completed In time, Ba^ed 
on our observations' oJ.;,the time spen^^ by the Sheppard staff.^: 
in Gompletlng earl ier resspifS , the task of finishing the 53 
POI htiurs of clinical and basic sclen6e materials plus the 
remaining pases in the series of simulated pa tlent encounters 
would have ^requj.red lowered standarfls of quality, more supertT- 
fic^l- ebvera tbples, .and even moi^e hours of Overtime 

work per day* than had already been logged.. 

^ The extreme tim'e pressure to complete the POMC lepso^^ 
i posed a continuous and vex4.ng problem to the prS^Ject staff* 
here/were a nLjmber of underlying factors to this plight^j 
not the least of wrtich was .an unrealistic expeatation at^ the 
start of the project as to the t^mf required to produce a 
given body of materials. In ^the original request for train- 
*^ng research (see Appendix' A)^ the project initiator assumed 
a production rat'e of 3^0 hours of author time to research^ 
develop, and test each studenb contact hour of CBE materials, 
(ThiB in itself is not necessarily a bad. initial estimat'e,) 
He then wertt on to compute a total time requirement of 60',000 
author hours to prod.uce 200 hours of material andj assuming a 
three year project and 2000 work hours per man year , ' deter-^ . 
mined that a manpower requirement of tkn authors was es^tab- 
lished," Agairij^ there are no problems wit' the fundamental 
arithmetic 6T^ these oal cul at ions * .However, this analysis 
failed tOrtake account of othei^ non-authoring duties/of the^ : 
staff and, mor e imp0rtantly>/ the^ f ac t that the proposed pro- 
ject milestones only all'dwed 18 months for development prior 
to the st^rt of^'the evaluation phase. Clea^rly a staff of 10 
should .not have been expeated to produce 200 hours of new 
mat0.ri|l.s in an 18. month period.^.^he ^job was too big for 
the number of people assip.ned to the project even if there 
had been no hitches. 
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Unrb^^tunately, the effort wa§ plagued by several tlm'e-. „ 
consuming problema in'/its Inlttal^t^ges . First, the p^jict' 

- init^lator left the prqjecfe; JuH;|et^|he tltnl it wa^ offioifflly 
gettlnR underway. Since^/he •was^^|h#6rl^inator of the POMC 
eoncept at Sheppard, his dep'artut^e left the PLATO staff to 
strwggle on its own in ihterpretlng and redefining the pro- 

• Jeet ' s Roals , » . • 

■ The definition process 'was comj^llAated by the staggered 
arrival of key staff members. Although the, project-approval;- 
. d-ate was November, 1973, ,;tfe>e phys loian, chosen \o serve as 

subject ma'tter specialist and curriculum planner dld^not Join 
, the" staff until,;July, 197i|.^ Many importa|?t decisions about 
the curriculum were postponed until his arrival, and .because 
^^the choices he made amounted to a change in the project's goal 
* ^^5^ F^ianning. an4 lessons developed to .that point were not ' 
A^^ilized. Even after 'th^.^physician^s . .arrival, there'^was. a . 
:shortage ..of manpower for l.esson development. Four of the ten ' 
staff members,.iaentified had yet to join the project, with 
the , last„-not arriving until November , '197'4 . ■ ^ T 
^ : Perhaps tli^' next- most tangible Ja^tor underlying the 
faiiurE't,© cWpl^te the POMC was the decision to'use'^nly 
one fourth o"f the PA class;.ln the .ex,per imental, group . Realiz- 
. ing that an n oM 16 was dangerq'usly small,- thi staff decided. 
'"■ to replicate the- experiment wit^^-'-a second ' cl#ss ' of PA 'studeRts 
to-start- the sequence the fol-lowing trimester. But in order 
■ «f or 'two, classes to -hflve time to "complete' the entoire POMC' % 
Within, the schedujea rfevalUatiorfi peridi^,, it was neGessary to 
move the stprt of the implementation £ptward/to July , 1 975 , 
reducing the time^ avail able for ^ lesson development > by six 
mor^hs. The scope of the task had been reduced somewhat by ^ 
^"^^4"^^"^^ ^i*^"^ Pls^s to develop 1S5 POI hours of CBE materials- 
rath^^pthan the 200 contact hours originally proposed. Meven- 
the.lesV, the task .was still to© large to be. completed in view 
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df^ the ■'I^it:f arr jy tl of . th 'Staff and the severely compr:essed 
/itime schedule forced' 'upon ; them by the sch^uling* constraints 
'of the PA Btudents. ^ ^ ' ' ' , ^- ' ^ ' ' 

There were other factors which had subtle' but. relal 
'effects on the project's rate of lespon^^Rfo^^^^^f*' 0^® of- 
the more .important df' -^ese wa§^ the fact AH^^^mWrtJ of the^ 
authors were considered ■; to be basic scieritlsts'^hav Ing a well- 
defined ^ubject matteJ^'.speclalt y C e . g ^ J chemistryj anatomyj 
pharntaci^>^;cSlJ'nical ^ Although each wa^ a. subject . 

matter expert in his own rights -most lacked the: broader per-^^ 
spectl ve ^ afforded b^y^bli.aic&l experience. Placed as they 
y^ere in a situation of trying to present -mater lal in a clini- 
cal context ( i , e . J il lustrating its clinical relevance), these 
basic scientists were at -a real disadvantip,e • They had to^'^ ^ 
depend, on the physiclaTT and another staff membqr^who was 
a' recent graduate of the PA cour^^ for assistance in this ' 
regard, Redausa. the^ basi^ scientists^ had limited^ expi^^'ience 
ih m#d;ical/olinical ^m^tt/rs reljatlng^to the resprratory sys- 
tem, they had to spetid ^pX^rea^ deal of time in 'r^seaVchirtg > 
'and organiziag -the topic^s they were -teachlnR p and hence they ■ 
were unable to develops m^terials^ as rapidly asvthey/oVjierwise 
might have. This problem came about because the dbjectivesa 
of ^:t,^e pro Jeet^^^re*^ changed afttaTT its inception p although 
staff was seieoted and hired/on the basis of the Qr^^inalj' 
goals* r ^ . r ^ ^ \ r 

other problem^ hind er irig tH^e lesson development pr©cJgss . 
inclMjded a ^general . lack of experience/training in instructional 
design procedures J the fact th-rtr-oivly few .'staff members had 
taught , portions of the .PA courie bffone,-only three persons 
had previpus teac hing ex p-er 1 en ce , and none had had previous 
Experience in designing* a curriculum* tt might also b^/said_ 
that talents, of some staff members we'V^e not fully' uti-li^ed ir^ 
lesson production , ^ *the Instruc tdona^l fly.stems pevelopment 
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specialist, th# illustrator, and one'of the experienced oomf^^h 
' Py^ei: V^^Rrammers proVablj could ha^^^been Aitilized more- 

fuli:y,^n an overall team eff^ " . '■ 

In summary, ^he task of d^velopin^ and irnplementing 
suchjla^Re arnounts of new curri*culum material,s was; hiBhly over- 
ambttiou^^a.nd.,u.nrealistic givan the manpower ■'liTnitatlort^ ant^ 
■ttm¥ dorts^C^intl .lmp^ed^on fcHe' 'pro4«ct . ProRresI was further , 
l imited .t^,^staRrered arrival of staff , ' thi change in 
project #^ls'lm'pliiH"% a V^int^T'fi^^ the %)MG^copcept , 

^ the fact fchSt the basic, scifintists' subject matter expe„rtlse - 
was rlulllfled by the decision to focus narrowly- o,n one organ V 
System, and failure to utilize fully- the . talent's of , the , avail- ' 
able staff. It was only throuF.h much hard work and mapy^over- 
time hours that the staff was able to complete as mucfT cburse- 
ware thev did.^ . ' - V \ : 

.nif£lpulty_ of -Deflri lng the POMC " : , ' 

' f Ije tfhlp^f'^aj^ the She>pard pro Jsfl^/ a . inWi 9I 

phase was ^that ^ndtYniueh was learij'ed (or at least not>iuch:was ■ - 
documented) about fte nature, of the^ POMn^ concepts and i.ts 

. i^'^T^^.^ curriculum ^haj: was. ^ 

i^iplementey was^tru^ly ^problem oriented, ^ ^ - ^ 

The person^ most knowledgable about the or Iginal concept 
fof fi^^probl^m-^oriented reorganization of the Physician Assi s- 
tanf cuPrftulum/ the project initiator, left the military 
just at the tlm^ the Sheppard CBE project was given officril 
approval ,\ iThe primary document which we have available d.esr 
cribing his concept ion .of th^e POMC is the Request for Traini^ - 
Reseir^.Csee Appendix \|). ynfor tunately , becaus'e of its brev- 
ity that document, does little to'edlfy the reader regarding . ^ 
the Original view of ^what^ would constitute a POMC, A later 
doa4jment (22 ■ June 1973) outline^ a reorganizaiion of the .eptirt- 
first year of the PA program under a group of 24 sets of " :/ ■ = 
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ratient compl!^in ts, :/ Under schema, the problem presented 

-'"1 ; 1 • ' = ■■ * ■ ■ ■ _ ' ' 

by the patient would -serve ss-^ m touchstone for Instr^jctlon in 
basic soienai and clinical material relevant to that partlo-- 
ular set of lij^mptbms ^ ^ . ' 

By the fbime of tha February^, ^197^ , planning session at 
the PLATO staff and other SHCS representatives 
riot to convert the entire .PA currl^culum to a 
^ted ( syraptoi^-oriented) format ^ citini? the sheer 



Shgppird AFB' 
had dec ided 
problem-or ie 
'S i ^e*^of jth^ 
Instead J the' 

the POMC (Kinball, ^lote 1) 



tiasr as^ beir^^yond the^scqpe of the prbje 
I adopted the following ""working definition 



A prqblfm-or iented .-Gurr J/culurD *ls the groupinf^ of course 
content lunder clinical proDlems^ and the presentation of 
materiall In a clinically-^oriented^ model such that the - - \^ 
eli^loall rej^evance o.f the course contint Is stressed ^^In ' 
prdeH^tol develop problem solv ing skills^ patient manage-"" 
ment and 1 invest tpative s4<il^Si ind^ th^^SoIlection of a > 

Included in ;tfiis >^eurr4ci^ urn Mt^e certain baai_j5*j^cience 
r^atelf*^i'^jls^ that^^:cannot b,e > d;ii^eptry reflated to clinical 
_ DroKle^s,^ Fhls material ' wlll^^ye included in the^curricu- 
^-^^lum as prerequisites to brifig students up to a minimal 
acceptable'- level of knowledge. To the maximum extent 
possible this mater ial wi*il be prese.ntftd in such a way 
as to^ contribute to the development of problem solving,; 
^skills through the use of inquiry learninR techniques, 

A\ passage from a later docufnent (dated August, 1974) 
reflect^ th6 staff's decision to focus oh topics relating to ■ 
the resplratoYy sjfstem* While it aip consisten»t w|.th the work- 
ipg defin\tion of a POMC quoted abovej It provides some addi- \ 
tio^^al insight *as to, the project's evol^inc plans. 

^Hatitnt problems (i,e, the symptoms or complaints for 
whidh medical attention is sought) will be' defined 4hd 
used as*a point 'of departMre^to teach the basic and 
clinical sc ience material related to the ; Respi ra tory 
■ System . , ■ ^^^^ I ^ ~ 

This v^ill lead to studenu-com^uter in tenac tlop^ to solve 
a Jyar iety ,of clrnical problems (i.e. exercrses in 'differ- . 
ential diagnosis and the.rapeut ics) . 
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;v : ^ Thusj the curriculum will be composed of a number *of 
V 7 -.Hnt^^ but potentially free-^atand Ing lessons deal- . 

ing with the basic and clinlc|!l sciences as well as sim- 
ulated '^patlent.encpunters , Th^ student^^^ili haye^ccess, 

* to basic and clintcai sclfnce review ma^rlsl^from ^ ^:-^- 
anywhere within= the lessan^Sequence. ^ 

The last portion of th^is s^rtiment emphasizes, t^e staff's 
expectatl&n that stojdlnfes /would ^e able to branch^back to 
^s^udy or review basic science -material relevant to speMfic 
cases . ^ ^ * ^ 

A September, 1974, document gave the following elabora-. 
tipn* of .the group^ s interpretation of .J'problem*.or ientatlon" : 

The term ^"p^oblem-orientsd iurrlcijlum" has been taken to 
mean several thinPgs: a) S^linicially relevant curriculum 

f heavily based on case despr ipt ions , usfng t^ose both ar. 
a pQi^jQ.t of departure ^d Is.a point of referend^^for 

' teaching the associated medical material, .rhis naturally 
implies (2) an integrated/^, approach ^tiicluding the areas^ 
pf anatomy, mtcrpanatqmy bioch%m,istr-y , genet ias ^ 4mmu- ^ 

% Hology , neuroscience, physiology, . pa t-hology/pa thophysibL- 
ogy.and ^cllni|al mediqine, pharm^c%l.o 

and|labQnato,ey tfchniques. In ^addity.o^, .there i{ill be 
^. practice in s^^^V in^ cl in leal problems 

and treatment plans for simulated oases) , 

Yet another__descr iptlve statement is given in the POMC 
Implementation Plan. Although the entire plan Is included 
as Appendix B, the relevant section is restated here to aid 
comparison : ^ ; ^ ^ ^ ^ - 

A' problem-oriented medical curriculum is defined as the 
grouping of bourse materdal under clinical topics whe.re 
the •cllnioal^ relevance of course contend can be stressed. 
Emphasis is' placed on developing problem solv ing skills 
in^ diagnosis ' and treatment of disease, 'lo facilitate V 
.. development', the POMC has been 'divided into t^o Darts. 
, Part I presents the^.* relevant basic saience qndX clinical 
material In .an integrated ;tnanner supported by illustra- 
tive case^^aterial and using clinical subjects as point 
of departure to teach basic sciences. Part II will v 
/consist of "Simulated Patient Encounters" which will be 
approached as unlwowns by the students. Diagnostic and 
treatment skills mk well as clinical problem solving 
skills will be amplified In Part II. 
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All of these ■'sta/tements are .fairly similar , but the subtle 
dLfferences may be particularly Important in vlew/of later ^ 
author cdnfuslon and disagreement over what a problem-oKl*en ted 

%X^- " ' ^ , i ^ 

J curriculum sh-Q.uld entail . As one author said, "There are as 

^ " . \" 

"many definitions of '_^obiOT;o as there a?e authors." 

*It is readily apparenV^^^ the stafff did not prepare a 
carefully-stated operatidnWi definition ^f^ the POMC which 
wpuld' aerye as a touchstone for ail ^ development/e^f orts * 

The Importance of cl inical releVance Is stressed in .al| 

.of' these Bt atem^isn^s , bLjjt they; differ as ±o what pl"ace clinical 
oases should ha-ve in the curriculum; T^Ab wasi" a substantial ^ 
'point of, author . d isagr eem,ent ^ Authors^ did not agree on the 

:qupstion of whether individual lessons, must necessarily include 
cases (or clln ical ^ref erences) , or even problem-solving > 

eleffientt* C ' ' I 

-* One author 'thoup^ht that a curriculum whlcl^ included^ 

. patient simulations could thiJS be qonsidered^ "problem oriented" 
regardless of whether other lessen^ in the curriculum centered 
on clinical cases or conditions. According to this criterionj 

.Sheppard's curriculum was indeed "problem or iented" , as would 
be- any others which include case simulations! ^ 

Other authors thought the curriculum should^, proceed 
backward frop clinical abases to the basic science material 
necesa^ry to di^aB^nose them, fina>lly ending in havinR^ the - 
student diagnose simulated cases,. 

i In a thorough review of those lessons known to have been 
Studied by PA students, we determined that only about seven 
percent included' clinical cases or simLJlations The project 
staff ^utlinedithe following criterion for a "problem-oriented 
curriculum in a September, 197^* document: " . a clinically 
relevant^ curriculum heavily based on case descriptions, usinrr, 
those both as a point of departure' and as as a poin^t of ref- 
erence for - teachinii the associated medioai material ...". 

{ . ■ .. 



Accor^lnB to thisr st^andard , the Sheppard curr iculum would 
'probaWy not be considered problem oriented beoause such a 
small fraction of lessons fit this description, 
. '\ Several factors co'ntribute to this/ problem. Firsts 
changes. in projpct admlnistratidn and the delayed arrival of - 
the senior content expert (who exercised authority over cur- ' 
r iculum definition and 'development) ^ resulted^ in; considerable 
rehashing and^ revart^lng of both the definitiy *of "problem 
orientation", and th^ implementation ' plan itBelLf. ^For Example 
■in the summer of 1974, when ^ bhe^^ filial group of Sheppard stfff 
visited CERL for tr^ning in- the TUTOR 1 anguage , they repor'ted 
that tb^ definition of ^problem orientation" was still being 
deternd.nedy ' This 'was nearly a year after the Inception of 
the She'ppard '^project-, _ ^ 

Second, due to the lack of firm administrative policy or 
control, authors exercised their own Jud.gment over what "prob- 
lem orlentad" meant, and how that related to their own lessons 
Consequently, there is little cu^rifculum uniformity either in 
term^s of ^problem orientation"' or in the approaGh to (or 
necessity for) teaching problem-solving skills , ' ' 

In a memorandum sumnarlzing the discuasions and decisions 

*made during the Februany, 1974, planning session, the acting 

project di^rector said that the " * primary objective of ' the 

projeofe is to reach' sofne cottcluslons regarding the instruc-? 

t ional ef f ec tivene^^'^pf I the 'co.^cept' of a problem oriented 

^ciar^r iculum" (Kimball , Note 1 ) . / AJudgirig from the variety o€ : ^ 

- ^ ■ . ■ 

perceptiofis of what constituted a POMC, the "concept" of a ^ 

problem-orSrented curriculum was never clearl^ defined • Jt is 

our belief that, in the absence of unambiguous criteria for* 

ddterminirtg" what ' would and would not qualify as a POMC, the 

question of the in&truct ional effectiveness of the POMC * 

"concept" could not have bean determined even If all lesson 

materials had been completed and properly implemented* 



Llrnit atlons o f t he Evalu ation Plan^ -j^. ' ; , 

A fourth^ major ar^a for analysis among the outcomes of 
the POMC phase is that of the plannlnB and execution of the 
evaluation,. Since a m.ajor goal of the proiect was to evalu- 
ate t^e concept of a. POMC,: it is^ important to examine the«/ 
plans ^and procedures that were. to be applied. 

Several points regarding the evaluation plans and staff- 
ing are noteworthy. First, there was a shortage of evaluatio.n 
expertise on the .project staff. . Th# person who was designated 
as^the' eval^ator 'jvas a clinical psycholof^lst who^ had come to 
the. project with the expectation that he would \work as an 
author on those curriculum' topics having to do with the ' 
psychology of patient, car r . ; When the pro,] ect '^s^ goal s were 
defined more narrowly in terms of respiratory topics, this 
individual was assigned the responsibility of dlisigning ahd 
implementing the evaluation. He had very little formal train-, 
ing in data-based evaluation procedures, but was, of course, 
well trained in observation and interview techniques. These 
skills proved quite useful in diagnosis of student complaints 
land in determin ing ^the ir reaso^is for^ asking to be dfopped 
from the POMC cond it ion , 

Perhaps becaus^ o^ the relative lack of local evaluation 
expert isa, ^ the Sheppard project staff had the continuing expec= 
tation that represent at iv eV, of other agencies would supplement 
their own evaluation efforts\ In par t i cular , it was believed 
from the outset that ^AFHRL wg/uld take an active role in struc- 
turing and car rying out the" evaluation , According to the 
Research Agre^ement^ AFHRL project mon i tor s ' wer e to assist in 
the measurement of, student and instructor attitudes as well 
as student performance^ This expectation was reinforced at 
an August, t97^4 , evaluation conference . hosted by AFJiRL. . Fur = 
thermore, it was learnerf at this conferenoe that ARPA had/ 
contracted with the Educatidnal Testing Service (ETS) to con- 

on 

' O i 

* 
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dufct an overatl evaluation of the use of the PLATO IV system 
at ARPA^supported military training sites (this responsibility 
rwas 'later "fe^ven to CERL's MTC and PEER groups. As part of 
its evaluation role, Efs was ^ prepare a list of ellnieal 
problems and to aSsistliv the measurement of clinical 
'problem*solv ing skills^ These hopes were later made more m 
sppciflc, as can be seen in the POMC evaluation plan (Appen-V 
dlx C). Aoaordlng to this plan, ETS was to provide^ items 'for,""' 
an examination covering respiratory disease* If Insufficient 
Items were available from ETS item banks, yet an<|ther outside 
agency (the University of Nebraska PKysiclan ^sistant program 
staff) would be called upon to supply Items* ETS was also ^ 

^ expected to provide some ."simulated patient encounters'* in 
paper-and-peno 11 form to be used to measure clinical problem- 
solving skills of both the TOM€ and Control group studerits. 

It is clear from Appendix C that the Sheppard staff was 
blanking heavily cm support from outside agencies and that 
they were willlngHo assume that these . agencies would be able ^ 
to carry through' on their, assig^ned responsibilities. It 
should be noted J however, that^at the time of the preparation 
of the. evaluation plan (March, 1975) very little work had 
been^corripleted toward the assembly and development of the 
measurement instruments.* . This was in spite of the fact that , 

^ the evaluation period was to begin In July;, 1975. 

The evaluation plan itself called fffr nine ^studies , 
These were: (a) a cdmparison 'of POMC and Control^ groups ' aca- 
demlc knowledge (of respiratory material), (b) a comparison 
of their problem-sol ving skills , -(c). examination of the POMC 
group's att^l'tudes toward computer-based education, (d) moni- 
toring of the POMC group's performance on lesson and block exams 
(e)-(g) studies of authors' performance, att itudes/"opini©ns , 
and . char acte^stics , ( h) summary of costs, and (i) a study 
of the reliability of the PLATO IV' system and terminal's . 



3d 



32 



H^ad they all been co.mpleted , these nihe^ studies woul<j 
have addressed worthy evaluation goals . However , it is doubtr 
ful whether many 'of them could have provided much general- 

,/lzable information. There are several reasons for this 

^ assessment . -. ' ^ ^ " ' " ■ ' . ■ - 

In' the case of the comparative stud'y of academic know- 
led^e, for example , the sole basis for comparison was to be^ 

; a pomprehensive tnst over basic science and cliniGal concepts 
relating to the respiratory system and its diseases. The 
plan called for some empirical development work. in the con-, 
struction of the test, but it is n^t clear that sufficient 
work cou-ld ^have been done in the allotted, time and with the „ 
available subject , pool to assure adequate reliability arid 
validity for the Besi red, pur poses , When such heavy reliance^ 

lis placed dn^ a single instrument, it is essential that suffi- 
cient development effort be invested, ' A 

Another set of concerns underlying the academic, compari- 

^son study ig tha*^: it was to be conducted under highly reactive 
arrangements. Because the Iniplementation constrain ts allowed 
for such a^small ri, the POMC group was treated mufch differ- 
ently than the Control group in regard '.to scheduling and test- 
ing, ^ These reactive arrangements created addational stresses i 
for the/ POMC 'group which may have interfered with (or enhanced^ 

,,that group^N^ learning of the target material: hence, the ^ 
rQaotlv^e arr^gements represen^^^onfpund ing f actor.s%whiph 
would have ^made i;iterpretatl^ of- c nesults quite 

difficult. Other potentially conffaminatin)^ inf ItJfi^^e^ "can ^he / 
traced to the srnall-grou^ study patterns followed by.^the PA*". 

'students,* It is likely tha t *POMC^ and Control group students 
Studied together in the evening hours and on weekends. If 
^SQ, it is qui te , possible that' th^r^fe^^fas a sharing of notes, 
recollections, etc.. which may have* aated to "wash out" any 

differential treatment effects. Although ItUs reasonably 

certain tha t Contrdl-^ group .students did not 'actually study 

■ . ■ ■ :■ ■ ■ . ■ ... ■ 
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: POMCf' lessons; on the .PLATO system , ihey may 'have had some 
vicarious experience with them through theiVfriekds in the 
POMC Rfoup, It is even rtore likely that the POMC", students 
may have Studied lecture notes for. those classes^ th^y had 
/"misused" while attendihR POMC sessions in the PLATO classroom 
At any rate, without ad^equate/cbntrQls: to assure the "purity" 
of the treatmenC cond i tion^s / i t . would^ be difficult to inter- 
pret, the results of the co.rnparison or to generaliM^ tliem to 
other settings* Furthermore, because the POMC was admlrfis- 
tered entirely on the PLATO system and the Control ourricOXum 
was presented entirely by traditional lecture approaehes, • 
there was a complet0 confound ing between strategy (-4,e. ,^ 
problem-oriented versus traditional) and mode^of. ^nes"fentat ion 
Had one group outper fbrmed another on the academic. testy:;lt . 
would be unclear whether it- had been because of differences 
in curricular approach or differences in' presentatlan<medlum/ 
This PLATO-POMC'' con found ing Was a pervasive pr,obj'^ and one 
which was. np^t; addressed by Sheppard' s evaluation plan, j 
Of the^i^ rem fining eight studies that were planned, dnly 
those dealing with authors' performance , attitudes/opinions,' 
ana chdiraf ter 1st ics are- singled out for comment here. As a 
whole these studies represent legitimate areas of inquiry, - 
but because of the complexity of such notions as "author per- 
fprmance'^ and "author attitudes"V they ar^e very d if f icult , to. 
pursue systematically. For example , author penfo'rmance was 
to be measure^d by Judgments of lesson quality , supervisory 
ratings, and peer ratings. The dimensions of TUTOR language 
programming ability^ lesson design efficiency A and creativity 
-were to be the Drlmary ind ices of ^"per formance!' , xat^d it was 
to be the .task of AFFIRL researchers to develop suitable scales 
and measures. To our knowledge, thi^s particular study ^as 
not ^ser iously^attempted-"perhaps because of thp grea^t diffi- 
culty ir> defining and measuring these dimensions , Likev^^ae, 
we know of ;no Significant effort to measure author attlC^o^es, 
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A number of aptitude test scores and other data were assembled 
on the author group to identify, certain key characteristics, 
as it was hb^^d that these coiild be used to predict what 
kinds of individuals should be selected as au.fthors for future 
projects. However, because these indi v id ual s had already 
been selected and were functioning on the Job, it does not 
seemvlikely tfrat they would vary widely' in ea si fy-measur ed . 
dimensions. Even if some relaticnshi.ps had bee-n found between 
■ author characteristics and performance in this aelected group, 
they may not be valid for choosing from the general population 
of candid^ates. goal of this study was a w6rthy one, but 

it is not clear that much of prediative value cyfiuld have been 
lear^ned fronf such a small sam^e of auth^ors.% Perhaps this is 
why the results of the study were not reported by St e in kerchner 
.et al . ( 1977) . ^ . . ^ 

In addition to these criticisms of what was planned for 
the evaluatio\ we can also point to some areas of omission. 
It is clear that the staff believed their pr imar y ^ noa 1 was 
to evaluate t4ia e f fee 1 1 venes^s of the POMC; they did very 
.little tso evaluate the effectiveness of its ^nTfj^l eman t a^io.n . 
Because of the many problems in defifhinR and .developinp; the 
new curriculum with a new medium under tifrht time constraints 
and the ri^uditi-s nf the PA course, the project was really 
one of nttemptiru; to initiate a iuimber of innovations in a 
hif^hly traditiunal and structured envlranment. One of the 
^^pr imo tniM^ets of opfiortamitv in snt^fi a ^settiru^ is the- docu- 
:nentnt. i^on of nrocedtn^es .usnd in instal 1 irij^ the innovations . 
In the POMr pvalnntio^i pi nti , fiowevtM^ t here were no [M ovisions 
for tlescribuu^ the oomm un i n a t i o n s betweeti the prU)Jo(H staff^ 
and member's of tho PA staf[\ Fu r t h e r mn rM^ , St e 1 n ke c lin n r et 
nl. (1^)77) did not t^eport (^n tliose imfHM^tnnt transac t i (uis . 
The liltelihniirl of i\iillnK to examine unanticipated nntcoines 
Ms ono of tdie major il t^a wi) ac l-:s to a pr cnii'd i n at. e apprMwioh t.o 
oval u a t i (Ml , 

s 

\ 

4 ^. 



35 



. "A second area If Opportunity that wa^ not suf f ici^ently - 
tapped in the POMC evaluation was tHat^of determlnirig/ those 
military training application's for which the PLATO system ' 
would be particularly well suited. The primary goal of thi 
project i^s to implement and test a /pfWdlE^ype problem-or i^ented. 
curriculum. The PLATO system was treated aB a iv en--a rned ium 
that ^had already been orov en^^-r a ther than' fts ^an important com = 
ponent of the ov er a 1 inno v at ion , It is true that the plans 
provided for an assessment of costs and rel iabi 1 it y o^- th# 
PLATO system, but they did" not provide for a determination of' 
its strenp,ths and^ weaknesses for medical ediicat ion , One 
result, of^ the staff ^ s ^seeminR a^^ceptance of PLATO as an estab = 
lished medium is that they implemented all POMC instriMtlon 
on the PLATO system. For example, they did not differentiate 
between instructional uni%s* that required ^ large 



u%s* that required ^ large amounts of 
lit did not--all were created in th^e 



interact ion^ ^nd thoie thfit did not--all were created in thV 
form of CHE' 1 essc>n.a. In contrast, it shoulri be r;ioted.that 
the plans^for Phafee II (see Appendix Dj called for a muc^h 
closer exdr^ination of the medium and its suitability fbr use 
\nth var£q|iis types of students and instructional tasks. ' 

In summary, the eval ua t i o n = o f the POMC phase reflected r J 
the constrain ts ^ imposed by.Vthe PA course and the implement 
tv:^ t ionV'pl an. adopted to accommodq^te %hose constraints. ^The - , 
evaluation ' ^in was concoLv^p^d to^fit tHje situation rather 
than definii:., the sltiiation tg^ ease the .evaluation prqccps 
and incronso the usefulness of its findinr^f^. The. staff put ^ 
tOfU'tlicr' a plan' that fit ttie c It^ciimstances , but tiieir joli was 
made fnor'e difficult by tlieir own inexperience and their result- 
ing dopondenco on othnr' af^encics. Furthermore, because of^ 
th^^ soverc conntraints of the course and the narroiv definition 
of evaluation tarr^ots ( o . . , p f f ec t i v.en o s s of the POMC with 
I i td J emphasis on the transactions inv()lvnd In its implement 
ration), the KcncM- n 1 1 xab i 1 i t y o th(^ evaluation ?'eMnlts would" 
have h^MMi lifnit.c^cj oyvn if idio I'OMC phnsc had not boon 
t cM^'n 1 nat <m1 , . > 
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Relations b et ween P LATO sta^ff and PA ^5;ba_f f . A $ indicated' 
earlier, the communicatdons between the PLATQ staff and the 
PA instructors were not a target of the SHCS/AFHRL evaluation, 
Beca'uie CE^L did not ha:^e access to the PA inWru^tors to 
conduct interviews, MTC ' s in form'at ion. about their^ attitudes 
toward and experiences with the POMC phase is also limited. 
However, through our discussions with the Sheppard staff we 
have gained the, general impression that communication with 
the. PA in^trMJctors tended to be sporadic and superficial. 
Many of the FA instructors appar entl y perceived the POMC/PLATO* 
project as beinfy a temporary and experimental ar r angenfen t' -- 
having, no Jong-term impact on the operation of the PA program. 
Consequently they showed little interest in the^ project and 
maintained a neutra^l or mildly negative attitude toward it. 
Initially ^hare was'^to be a represen tat ive 'from the PA staff - 
present at the PLATO project staff meetings. -^Likewise, ^th^ " 
PLATO group was to send representatives to the PA staff's 
meetings to act ^s liaison. However, the importance of 
mc^intainln^ good .communications between the few.o groups must 
not have been fullv appreciated, because the frequency of^^ 
"Joint .attendance" ati meetings Soon dropped nearly to zero* 

N o t o n 1 y xi , i d ' t Pi e f n r m a 1 c h a n n e 1 s n f com m M i c t i o n r e m a i n 
unrUMU]t 1 1 i zed or Ineffectavp, a numh^r of informal' approaches 
r\]3n had lioiited juccos^. A test gradW avei^ager and on--line 
J \p b iiijk w ^^'1 s - p r' e p a e d by a n p a f t h ^ ^ P L A T 0 s t a f P ^ t o h e 1 p t Pi e 
PA 1 tist rMuitor'af'wl^ii the .m^ecuhanXQ^^ of gr^ading^ ( mad^ OVore diffi- 
cult by t he 50 p;?1' n t e make-up t?'xams for J the ■ POMC^^rf^^Up^ - Tni.-^ 
spite of its apparent convenience, it went unused and unap= 
preciatcd by tlie i n strMjc tor s , Individual authors would often 
go to their CO un t c iM^a t ori the !^A staff and enlist his or her 
help in revlewlnf^ a lesson corresponding to a lecture. Though 
the PA 1 n st r lie ttH^ ^^ \^orc generM 1 1 v 'ccioperat 1 ve , they tended- to 
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view the PLATO materials as s nice adjunct but not as a 
replaaenient for the lectures. In a few cases there were 
somj disaBreements about the depth of coverage fcfiat should be 
c,iven. These arose out of '^the ambiguity of the course's Plan 
of Instruction and a ■hanging conception of the role of a PA. 
In s,eneral, however,, the instructors did not play a large role 
in determining the content and strategy of tbe lessons/ 

It is not" known how much the PA staff apathy may have 
affected the students* "attitudes or performance^ but it was 
probably not a positive influence. A more systematic cultiva- 
tion of relationships with the PA staff ' might have helped ■ 
generate a more favor^able attitude and greater cooperation. ^ 
On the other hand, g^^^en the f und amen tal cone ept i on of the 
project^ as an experimental implementation of a new and unproven 
style of curriculum, the PA .staff may hot hav e ,beer}v" won 
over" to an enthusiastic and positive outlook no matter what 
techniques were used. 

Intera ctions ■be twein MTC and Sh e^par d staffs. The MTC 
group (through its specially-designated^ subgroup "Shp East.".) 
was given a contractuai fole by ARPA to provide jupport to ' 
the Sheppard staff and to participate with thejn in develop- 
ment of the POMC.^ The sponsor had visualized MTC * s role as 
one of active paVtac ipat ion--particulaVly in those areas of 
the project where the MTC staff'^'s experience could lead to 
significant time savings (such as^ in developing, tlpe progranfs ' 
to implement the simulateid pa ti e^t^ encounter s ) , A graduate 
of the SCHS Physibian Assistant program was assigned by the 

cU^ tvith '^TC staff members as a' subject 
hoped that his pnesepce at CERL would 
enable the MTC staff to^iontribute directly^ to project 
planning aad deveFoprr^ent , ^' ^ ;/ 

Although workin^^relationships betwcBn the MTC and' 
Sheppard staffs were congenial and productive, there was 
a r^^luetanae on t;he part of the Shcpp|rd grpup.to give MTC 

■ , - ■ ■ : ^ ■ : ■■. 
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the role of co-participants outlined in the contract. This ' 
may have been partly due to co.ncerns aboiA MTC'*s lack of = 
specific .subject matter expertise "( sccentu;a4;ed> by the -early ^ ' 
departure of the. PA asr^igned to assist at CERL) , Another 
obstacle was the geographic separation between Sheppard AFB 
in ^Wichita Falls, Texas, and CERL in Urtfana , ^11 1 ino i s , , Th'us , . 
with some exceptions ^ MTC's role on the project tur^ned o.ut to 
be one of providing advice and sunport rather than one of \ 
direct participation. 

If it was difficult for.th^e Sheppard staff- to consider 
MIC as co-participants, it was even harder for them to accept 
MTC/PFER inputs and par t ic i pa t io n as evalljators on the project. 
The primary intent of the sponsor was for evaluation staff at 
CRRL to work to enhance the effectiveness of Sheppprd's 
efforts... It. seemed, however, that the Sheppard group ptrceived 
CFRL^s evaluation role at that stage as being summative 
rather than formative. They were uncomf or tab] e wT^ the 
notion of ^'beihg evaluated*' by a group that had previously 
been in a siJpRort role as co- par t ic i pan ts " , As a result, a 
considernbly less interactive r/ole was negotiated with ARPA 
for MTC and PEER than had been original^ly anticipated. " 

In general, thk impact of C^RL*s invol v ement wi th ■ ^hc^ >■ 
ShRppard nroiect was significant, but probably not as great 
as it might have been. Perhaps the greatest factor liFiiting 
MTC^s influence was the geoGR^aphicnl separation between CERL 

r. 

and ^Sheppard AFH. Other obstacle^^ to collaboration (e.g., 
differences bet^vrnen/'mi^J. ita r y ^nd cl vi II an^ met hodoiog ies ^ind ' 
MTC's lack of rnedica] siibj ec^^matter expertise) would have 
been 1 e s s ' t r o u b 1 e s o m e . h n d the t i^/o g r o n p s been in close r 
proximity, For example/ the Sheppard staff stated that MTC 
reviews of their lesson5^ would have been more helpful had the 
reviewers be^n located at Sheppard ( Cal 1 ^Himwich , lQ?7a)* 



' OVERALL ASSESSMENT 
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The most imporiant (or most visible) categories^ of'-out- 
comes of the POMC phase were discussed in the previous sections 
Of these, the| most tangible was* the move by mafi'y of Che POMC 
group studeats to be drppped from the experimental condition. 
The primary basis for their request was a d issitisfact ion 
with the implrementation of the POMC vis^a-vis the regular 
^portions of the PA curriculum. Being a minority of only one ' 
fourth of their class, the POMC students felt threatened by 
an implementation which required them to "miss" reetures 
attended by the majority of their peers. The make-^hlft test- 
ing and examination procedures were also a source of discon- 
tent. Another complaint was, that the POMC appeared to be 
fraf^mented and lacking in continuity. This, of course, was 
a further result of the perceived necessity to^ replace discrete 
hours of the regular course with new materials from the POMC, 
Some problems having an impact on st{jdent attftudes grew out 
of^:the time f)ressures on the staff and henae|^ were... only indi- 
rectly related to the implementation. These "incl uded an 
unfortun||e introduction to the project\^(in which privileges 

'were f 1 r st ^ of f ersed and then |iW4t1idrawnt. and prog ramming error s 
in relatively untested (and unvalidated) lessons. It is dif- 
ficult to assRSS the Impact of the- latter problems, but we 
bplieve them to be secondary in importance to the fundamental 
constraints qq the implementation Imposed, by the PA course 
Hself, , ■ ^ . \ ' ^ V 

Jhe second major categof y of outcomes was the\ complex 
of factors which impeded lesson develohment progress. We 
believe- that the, POMC Jfss'on materials may'not have been 
ready in time^or use even if the first project phase had not 
been terminated early. As detailed above, several factors 
G o n t r I b u t e d ^o this s i t u a t i o n , They include an unrealistic 
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assumption about the amount of materials that^coXUd be'/pro- 
duced in a Riven time period , the late and staggered arrival 
of strff members, a sever el y compressed time schedule due to 
the desire to rua'^.a second group of 16 students, and a change 
in project goals which-made it necessary for some staff mem- 
bers to write materials outside their own specialties, hence 
decreasing their efficiency. ^ - 

'Another clear outcome of ^the first project phase was that 
the concept of a problem-qrlen ted curriculum was not opera- 
tionally defined. ^ After the pr^oject initiator had, left the 
service, it was up to the remaining 'staff me^mbers to settle 
on a definition of the POMC. They never did. Although work 
on a definiti'bn be^.an in Februar y , ^ 1 974 , and continued through 
out that year, there was still' enough disagreement at the end 
of the first phaa#- that ape staff member stated that "there 
are as many definitions of ^problem oriented* as there are 
authors." In view of the - un r e soK^ed amb i g u i t y as to what con- 
stitutes a pr obi em-or ie n ted curric^lum^ it is not clear that 
Sheppard^s Phase I mate^rials qualif\ed aa a PORC. Had that 
phase run to completion, it would' have been viiff^cult to Judfi,e 
whether or not the concept of a prob leVi^or iented curriculum 
had been Implemented and tested. 

Furthermore, even if the ^student sample had not di^min^ 
ish^d to the verf^e. of extinction, if the POMC lessons had been 
completed j and if they had been truly representative of the 
concept of a problem-oriented curriculum, there werd several 
deficiencies in the planning and r?onduct of the evaluation 
which would havn limited the g en e r a 1 i z ab 1 1 i t y of the results. 

Concl us ion i 

It is klear that t+ie POmc phase of Sheppard's PLATO pro- 

1 ' / 

iect did vnot^ meet its obiectives. Unfortunately, the innovR- 

tive nature of the POMC concept mny have been more suited to 
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evolutionary development than to the systems approach that * 
was attempted. It is now apparent t,hat too little was under- 
stood at the outset about the POMC concept and how it should 
be impldmen ^ ed to Buide the project's development. This was 
not clear Bl the ^beginning , however, and the negative outcomes 
could hot have been foretold* In spite of its early termina- 
tion, the POMC phase was a worthy effort--a number of lessons 
were learned which provided a firmer basis for the follow-on' 
project. 
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REQUEST FOR TRAINING RESEARCH 

School of Health Care Sciences, USAF 4 
Charles N. MuHUan, Lt CoT, USA^', MC 

Sheppard AFB, Texas 76311 ^ ^ ' ' 

Autovon: 736-2210 ^ 4 April 1973 

Evalua^on of PLATO IV System in Air Force /Medical Training 

A critical shortage of. physicians* services exists In the USAF 
Medical Services reflecting a similar nationwide deficit. 
Traditional modes of health care del 1 very ^have been maximally 
stressed and found ^i nadequate ; new categories of health mah- 
power have been developed. One such new health professional 
Is the physician assistant (PA) ^ a person who because/ of spe- 
cial training and experience can perform many duties formerly 
carri^^ out only by physicians, 

/ ' 
Tho^allied health education of the Physician Assistant and 
th4 medical education of the physician are remarkably si^mllar 
.^fn content and scope, therefore most PA programs are modeled 
after traditional medica) school curricula. The instructional 
approach is characteristically one in which there is a horl" 
zQntal sequencing of basic medical sciences with clinical 
practice. The usual modes of teaching and learni.ng are lecture/ 
demonstration in group/lock-step pattern* The major difficulty 
with such a curriculum Is that the co^re _of medical knowledge 
which must be memorized l^^isually taught without a fundamental 
frame of reference, Basit madlcal sciences are often presented 
in isolation from their clinical relevance^ both intellectually 
and temporally. Thus, when the clinical sciences are ultimately 
presented J time-consuming and repetitious review of the basic 
sciences must precede the teaching of a new body of knowledge. 

The basic frame of reference missing In a traditional medical 
curriculum Is the core of clinical behavly which characterizes 
the problem^solvlng process. Its absence has heretofore made 
the application of recent advances In educational technology 
(systems analysis and design) difficult and j neons 1 stent . 

Two major ^object Ives of this study are envisioned.' (a) The 
research and development of a new, prob 1 em^or I ented , medical^ 
curriculum atlthe physician assistant level using an advanced, 
computer-based educational system and (b) a comparison eval^ 
uation of the Instructional effectiveness of this curriculum 
vls-^"vis the present traditional curriculum. 
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Advanced Research Projects Agency: To^aka avaijable the 
development of a fundamarita 1 jy new probTln^rlented cur- 
riculum upon which all allied health instruction patTon- 
wide can be constructed. This prototype would have ^he 
potential for the complete reddrction of all present ^ime- 
consuming, repetitious and costly instructional practices. 

Department of flefanse: To provide an instructional test 
bed for the new geTieration of military hospitals project, 

USAF/SHCS: To make available a comprehensive computer- 
based educatio/ial system for use In training and educate ^ 
ing allied health professionals In the USAF medical servide 
through the utilization of existing PLATO IV programs as 7 
well as connection with the tie Into other computer-base^^ 
networks 
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University of Illinois: To provid^e CERL's PLATO IV project 
with all the necessary requirements for the research and 
development of the types of hardware necessary to meet the 
unique requirements of the medical education system by 
providing the data base^ I nstruct iona T Systems Design, test 
students, and evaluation protocol. ^- 



6. ' MAJOR USERS: a. SHCS 



TIME 
PHASING: 



8, 



UTI LI2ATI0N 
CONCEPT: 



a . 



Other uniformed services 

Civilian educational institutions * 
NGMH j ' ' ' 

Project development: Approval data + l8 months. 
Te€t evaluation: Approval data + 30 months. 
Final report: Approval data + 36 months. 



Because the development of a new curricular approach Is an 
evolutionary process, 1 I ^ d 1 f f i cu 1 1 to foretell the state 
of the art at the time of Final Repor^l Nevertheless, it is 
conceivable that the man^madhi ne i nter f aca will result in a 
continued reinforcement of ttie prob 1 eip^so 1 0 1 ng process as 
the dominant mode of cllnicaj behavior. Instructional modules 
will be us6d to teach a basic core of medical knowledge as it 
relates to clinical problems. This core of knowledge will be 
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available for other allied health personnel to aid In under- 
standing their own roles in th^ delivery of health care, that 
IS, thair place a long the d iagnos i s = t reatment loop. Concur-^ 
rently. operating will be a continuous eva luat ion.of the re- > 
lative merits of the two curr i cu la , ' w I th and wltTiout PLATO IV. 



BRIEF OF 

THE PROJECT: a. Approach 

(H G I yens 



I (a) A basic core of ^medical knowredge for USAF Physi- 
cian Assistants exists as outlined in the Plan 
of Instruction, Course 3ALR91730. dated 3 August 
i972 (Appendix A) . ^ 

.' ■ • - , ^ 

(b) A bas Id^pattern of clinical behavior character- 
'izes problem- solving which is stra igiht forward, 

logical, and scientific,'^ 

(c) A charting system exists for recording and docu- 
menting the use of med t ca 1 knowl edge i n the 
solving of clinical problems* 

(2) Assumptions: 

(a) Advances in educational technology and Instruc- 
tional systems analysis can be greatly aided by 
a computer system In whlc^ [9a(i)(a) and (b)] 
(above) are integrated and coVrelated. 

(b) Such a system will be self-paced and learner- 
oriented. Evaluative systems will cons,tantly, 
assess the^fect i veness of instruction and the 
level of ^udent aehievement. Quality instruction 
and performance will not be compromised In de- 
ference to resources conservation* 

(3) Methodology ' \ 

(a) Approximately gOO student contact hours of new 
curricu^lar material will be developed ort the 
University of Illinois' CERL's PLATO IV system. 

(b) An Instructional Systems Des ign lAW AFM 50-2 
will be developed and the rough outline of a 
problem-oriented curriculum will be prograpned 
into PLATO IV. ^ 
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(e) During the davelopmmnt of this project, students 
will be eKpoisd to both currlcular formats In 
which PLATO IV may or may not be employedi The 
sul tab 1 1 1 ty and eontrlbut ton of computar-dss Is ted 
Instructlonr^wl 1 1 be evaluated as It stands alone 
and vJs^a-vIs the eurrlcula, 

(d) The attached management guide for a patient entering 
the emergency room confused or In coma (Appendix B) 
jllustrates how the problem-oriented approach niay 
be used to organize currlcular material Into data 
gathering (S_ubJ active Information and O^bject^ve 
^^flata), data syntheils (Aiiessment) and problem re^ 
ioluttan {£Ians) . At virtually every point along 
the problem^solvlng loop, branching Into the body 
of basic and clInFcal medical knowledge can be 
made; for example, whMe studying the data base 
contained In S^ubjectlve Information, the student 
can review methods of Interviewing, pertinence of 
questjons to be asl<ed, and so oni Whan reviewing 
the "Informat ion to be gathered un^er Objective data, 
the student can correlate anatomy* physiology, radio 
logy and techniques of phys lea I'examl nat Ion. Under 
A^ssessment, the student Is appropriately exposed 
to classic descriptions of disease processes which 
can be integrated with presentations I/i pathophysio- 
logy and biochemistry. When learning of the problem 
resolutjon phase (P^Ian) the student Is exposed to 
the fundamentals of laboratory medicine (cllnlca'if 
chemistry, microbiology , serology, and so on) . Add I 
tlonal ly, pharmacology is rational ly Interpolated 
at this point to describe the basic knowledge und'er^ 
lying medical therapeusls. 

It Is apparent *that such an approach stresses the 
Importance of a certain pattern of clinical be- 
havior and it is^ this critical area that will b^ 
subjected to intensive evaluation as to the roT^^ 
of CAI in d^^^lopl ng^ and maintaining desirable 
attitudes ari'd dispositions. 

Support required 

,(1) Personnel ' , ' 

300 hours author time (AH) are required to researcj^j, 
develop and test I student contact hour' (SH) , 

-J 

200 SH = 60,000 ah: 




' • ■ ■ . , 49 ' ' . ., 

i > I mill-year ■ 2000 hours. 

Therefore 200 SH - 30 min-years. 

^ three-year project, a need of 10 persons Is 
^ . established. 

, W PLATO IV tefmlnals; 20 (15 itudent, 5 author). * 
,(3) Dedicated teleconi lines: 5 

W Advlsor-eonsultahts at_tha Unl vers t ty of llllnolsi 2 

(5) Physical plant and equipment 

(a) 20 carrels 

(b) Adequate :a I r oondft toning to keep units suffi-' 
dent ly cool (20,000 BTU per tarfhlnal ) . 

(6) Installation and maintenance of PLATO IV eqLipmant at 
. * SHf S * 

(7) Training of ten SHCS perionnel at the University of 
Illinois In TUTOR language, currlcular development 
and Instructional capabi I Itles-of PLATO IV. 

10. SUMMARY: A new currlcular approaeh to the teaching and learning of 

probleni-orlented healfK care, iystems analyzed and developed 
in conjunction with, an advanced computer-based educational 
system, is described. The Impact upon allied health education 
. at the SHCS Is all-encompass I ng since such a curriculum offers 
an understandable frame of reference forjhealth care delivery 
as .It relates to the respongibi ritles of each training depart- 
ment. The research design will attpmpt to provide significant 
data comparing the effectiveness not'boly of (he differing 
curricular approaches but also of computer assisted instruction 
in health education. ' 



lo^'ni^ ^' Walker, H. K. : The Problem-Oriented System. MEDCOM 'Press . 
iy/2, 20/ pages ■ ' 
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f . POMC IMPLEMENTATION PLAN 



I, Referendesi Course Chart 3ALR91730 10 Feb 1975 

POl 3ALR91730 

Research Agreement "Evaluation of PLATO IV 
System In Air Force Itedlcal Tralrflng" ' 

II, Plan Objective 

The purpose of this pian Is^to provide for the orderly integratlDn 
of the ProblM Oriented Medical Currlcultfffl (FO!C) developed by the 
PLATO IV Davelopment Branch into the presenrrphyslcian Aasistant ' 
curricula. This Integ^cation.will be do^e in such a way that; 
^1, Disruption of the present curriculum is minimized. 

(a) Steps will be taken to insure that prerequisite material 

is taught prior to the time It is required as a basis for other material. 

(b) The logical flow of content in discipline oriented blocks 
will not be jeopar^rEed* 

,2. The integrity of the POHC is maintained, i,e, presentation of 
material is not excessively fragmented* 

3, No degradation In training will result. 

4. Evaluatioff of tha POMC is facilitated. 
III. Scope 

This plan pertains only^tO' the Physician Assistant ^^rse (Phase I) , 
3ALR91730. The time frame for implementation of this plan is Jun 1975 - 
Oc-t-- 1976 to correspond with the evaluation ^hase of the PLATO IV project, 
IV, Background * 

A, On 4 Apr 73, the School of Health Care Sciences gubml'tted a 
proposal for personnel research/to Hq ATC* The resultant research agree- 
ment calls for (1) the research and development of approximately 200 
hours of a new problem oriented medical curriculum at the physician 



assistant level, uaing an advanced cofl^utar baied educatlpn syatefliy 
*ajid'(2> the evaluation of the InsCructlonal effeotlveniss of this 
curriciU,™ vls--a^vls the traditional^ curriculum, /^^e project [co^btb 



'a three year ^^iod beginning In Noveaber 1973, Evaluation la to be , 

-- - ■■ f , 

^^ompleted by 1 J^y 19 765 with the final report to be rendered In 
Noveinbar 1976, . . . 

B« A problem oriented medical curriculum is defined as the 
grouping qf course inaterial under clinical topics where the clinicai/5 
relevance of course content can be stressed. Emphasis is placed" on 
developing problem solving skills in diagnosis and treatment of disease. 
To facilitate development^ the POMC has been divided Into two parts. 
Part I presents the relevant basic science and clinical material In -an 
integrated manner supported by illustrative, case material and- using 
clinical subjects as point of departure to teach basic sciences. 
Part II will consist of "Simulated Patient Ehcounters'* whichS^lll be 
approached as unknowns by the students, DlagTOstic and treatment 
skills as well as clinical probleA solving skills will be aB^lifled 
In Part II, 

C, The task of selecting the content of the POMC from the massive " 
amount of raateriai included in the present physician assistant course 
was undertaken as follows. Selection was based on the criteria that 
the clinical area would (1) accommodate an integrated approach to 
include both the basic and clinical sciences, (2) contain fairly clr-= 
cumscribed and clinically relevant subject matter, (3) provide sufficient 
subject matter to meet the project objectives, and (4) make the best use 
of resources within the PLATO IV Development Branch, Jointly exhaustive 
lists of possible etiological and body systems were tested against the 



" ■ "■ ■ . ' ■ ■ ' .:'•'■/•• ■ ■ 53 ' ' ■ ■ [ -J " : ' ■ 

J selfctlon criteria. The respiratory system ('jripl#atorf^liaaM) was 

■ ■ -^V . ■• . ■ , • 1 " ■ f . 

* selected aa 'the beat subject area for the purpoaefl of this project. 

~ •' / y J- I ' ] ' ' 

> , Dl A list erf baa Ic sciences ^lid apeclfic Icllnlcal toptca has been 

' v ' ' ' . I ' : . ■ 

, i .,<eyaloped and arranged In a currlcular at^ctute which defines the 

oaslc sequence and InterrelationahlpB Of the -aubject matter areas. 
Initial plannlng/prograronlng has begun In alght of the proposed 20 / , 
lesson development areas, i ^ 

V. Approach to In^lementatlon 

A. Definitions and Constraints i 

i ■ ■ . * 

1. Termlnais. There are 16 PLATO IV student terminals. Prime 
time (when the system Is moat stable) is from 0740 to 2200 hours, 
M?nday, Wednesday and Friday, 0600 to 2200 Tuesdajl and aursday, and 
0800 to 1200 hours Saturday. The total available terminal hours ds a 
constraint, since there are 64 PA students in any given semester, plus 
30 PCNP students who attend PA clinical medicine ledtures. 
, 2, POI Hours. For planning purposeB, hours,\are considered 

to be POI hours planned for a particular subject. POI hours may not 
necessarily equate to hours actually spent at the terminal by a given 
student* PLATO material at the lesson or lesson sequence level ie 
sa^-paced and variations in individual itu^dents can be expected. 

3. Classroom <C/L)/Complementa^ Technical Training (CTT) Hours. 
Since counter assisted instruction la designed to teach for mastery of 
the materials a PLATO IV lesson designed to meet a particular POI ob- 
jective will replace both the associated classroom -^ours and CTT hours. 
In coc^uting hours in the present curriculum to be replaced by PLATO 
material^ both C/L and CTT hours are considered, 
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B,. Hours Required lor POMC Haterlal. ' 

I, The*present PA curriculum has been eKamlnea to Identify 

those hours In 'which .the subject fflattet is essentially the same as 
POMC content and can therefore be replaced by the comparable POMC 
material, A detailed breakdown of hours ^by^PQI objective by semester 
is included in Attachment 1* A sij^^ry 'Is provided in paragraph 3 

J -. : \ ' % ' 

below. 

2* As anticlpfftadp some material In^ludfed in ^ the POMC does 
not have direct compdra'blll^y with hours in the present curriculuin. 
This material is limited to "simulated patlei t encounters" wh|(chMs 
expected to requl'te 24 stuflent contact hours* These houi^ are considered 
to be an essential part of the POMC for review and reinforcement. 



f 3, Summary of Hours 

Replacement Hours i 







C/L 


CTT ' 


Sub-Total 


1st 


trimester 


16.5 


7.5 


24 


2nd 


trimester 


20.0 


5.0 


25 


3rd 


trimei ter 


52.0 


20.0 


72 




Sub-total 


88.5 


32.5 


121 



Total 



121 

Additional Hours i 

Simulations 24 
' Suh-t6tal 24 

^ — 

I 

Grand total i 145 

4, in the original research proposal an arbitrary estimation 
of 200 course hours was designated for PMTO IV programs. The reduc^ 
tion in these hours to 145 represents the integration of an effective 



circmnscribed curriculum into the present course with the least 



^ disruption of continuity^ Thd BKpmMion of the PL^O hours' to reach 
■ a .closer apprexitaation^of the ^original ^tiaation^f hours would o^y J 
serve to dilute the"^ effectiveness of th^'PLATO project^ complicate 
the evaluation procedures and increasp the problems of integration 
into the present coarse* ^ ^ 

C. Planf'for In^lementation K 

1, sanple of 16 Physieian Assistant students will be selected 

from the freshman cl«s starting in June 1975 to serv^ as a^lot group, 

* ■ " ^' 

Sampling techniques developed by AFHRL will be used to^select the 16 

students. A<seco»d"san^le of 16 students may be selected from the class 

starting in Oct 1975 if that is considered desirable on the basis of 

evaluation plans and experleace with the initial group. (/ 

2, Students in the'sanqpla will complete PLATO materials in • 
lieu of the corresponding work in the current curriculum. They will 
attend all of the scheduled lectuieSj except those that have been 
replaced by PLATO- lessons , In addition to criteria listed in IV-C 

above, considerable concessions have been made In the selection of material 
to be included in the POMC so that discrete lecture hours of . the curreiit 
curriculum are identifiable* At the time such lectures are glven^ 
students would leave the classroom and report to the PLATO lab for 
cbmputer based instruction. Overall, the same criterion objectives will 
be met by all students, but the temporal sequencing of material in the 
problem oriented medical curriculum will not necessarily parallel that 
^of the current curriculum, ^ 
3p The additional hours required for the simulated patient 
encounters included in the POMC will be supplemental hours to be done at 
the student's convenlances , This will provide review and reinforcement 
of material covered in Phase I. 



; ; f 4. Since students In the samp Ig group will not necessarily be 
working on^ the same subject mt^tfr at thfe same, time as the rest of the 
class, BowB modification of the normal end of*block evaluation procedures 
will be required. POMC students ^uld take the normal end of block 
tests* butf questions cove rfiig courq.ework tau^t In the POM€ would not 
be counted in their test score. The PLATO IV staff will assume respon- 
sibility for determining student proficiency in Bie POMC subject^^reas 



on a semester basis to detWmlne whether or not the student is cotopetent 
to proceed* needs remedlati'onfl etc^^ The Pl^TO staff will also assume 



responsibility for* n^ded reme4iatiop in .fliose areas 
VI. ,Taska Remaining to be Done 



A, ^ detailed schedule must be developed for the present curriculum 
that Identifies specific subjects for specific lecture h'oursj and also 
identifies hours during which POMC material is to be taken by the sample 
group . 

Waivers from CQurse ddtumentation requirements and testing 
standards must be requested from Hq ATC. 

C. Any questions * Involving accreditation from the University of 
Nebraska for the sample ''group mUst be reaolVed. 



1 Atch ; \, 
Breakdown of ement Hours 

by POI Objective 



B.lock Paragraph Objeetlve 
VI 



vrii 



IX 
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6a 



lib 



11c 



lie 



6a 



lit, 3em - RiplaceiBlnEi 

poi mmm {%yi) 



Analpi tht prlnciplBS of Ehe . 
hetarophlli tist . - 

Iranslate the principles of the 
antlsEreptolysin-O 

I 

Cofflpire the prlnclplaa and)pro- 
ceduris of the febrili agglutin- 
ation tests 



Antibiotic susceptabillty test, 
Coiiipari two nethods of testing the 
susceptabillty of bactirla to 
iicrobial agents and intarprgt 
resulti of such tests, 



Freiint 
Hours 



Idetitlfy the clinical topography (4/2) 
of the thorax 



Lilt the steps in the Gran Stalni ' 2.5 
Acid Fast Stain, and KOH prepar- - (I/, 5) 
ationi and Hit the purposes for 
perfonring McrobMogicaTstains 
on cliriical speciaSi 

•List Ihe proper lethods of spicioen 1 
collectioni the pitfalls in the 
hMing of clinical specliBins, 
and relate the collection and 
handling of specinens to bacterial 
and viral infectiona 



2/J 



Subject 
Anitoif (the^ Etunk) 




1*3/, 5 Sirology 



Serology 



Speclien GoUiCtlon 



Antibiotics 

J 



m 

Hours 
4/2 

I 

1.3/.5 



1/.5 



1/J Febrile A|glutinitlon 1/j 



Itainlng procedures 2/. 3 



2/, 5 



6 



Block Faragr^h Obj^ictlv a 



IX 

cont'd 



9d 



6b 



Idsntify thi acid fait orginlsmi 
and thiir clinically ilinificatiE 
pathopnlcity 

Sglict the basic asiiiiptions of 
the theory of kinetic Boliculir 
gases 

Solve probliBs involving chanpi 
in the presiure, tiiperature, 
voluie and nuiier of molei of 
pses, • 



Preiint 
Houra 

m 



(2/1) 



Subject 

Clinical Correlation 
of Coiion Fathogins 
(Add F^t Orpnisn) 

The Gag Laws 



r 



ill ' 



2/1 



01 



16,5/?,5 



60 



i 




2nd Seiii - Repkcefflents 
POI 3ALR9n30 (Vol 2) 



Prasent 



POMC 



Ml mm ' . Hours 

Descrfbe the whanisffls Vor main- ' 8 Fluids ind Electrolytis 6/2 

tenance of fluid and acid^base . (6/2) (Add-base balance and 

balance .in the body fluid compart- , lat tests. 0. and CO- 

mm and deterniine the use of ■ ' transport ^ 

laboratory test reaulte^ in classi- ^ ^ ^^^^^^^ 

tying abnoriiallties, 

Describe the anatoray and eKplaln 12 • The Respiratory Systei 9/3 
the physlojogy of the respiratory 



sys tei, 



1 



Identify and detarniine treatinent 6 Diseases of Ear, Nose • 3 

of coiiion diseases of the nose. (5/1) and Throat ^ (N«se) 

Identify and dutOTine ireatMnt 5 Diseases of Ear, Nose 2 

of coinion diseases of the oral , and Throat - (Throat) 

cavity and the larynx. 



TOTAL HOURS: 20/5 



/ 



3rd Sib - Replaceiants 
POI 3AIR91?30 (Vol 3) 



Block Paragraph 
XVIII la 



Ob]ecEiva 

Identify the atmospharic condition 
that can affect the huMn body in 
flight. 



PEesent 



3 



Flight problanis 



POMC 



2a -n 



2a 



2b 



2c 



2d 



2e 



tha radiographic densltiis 
ed by roEntgin beasi* 



NaiDa the anatofflic structuraB dee- 
cribed on a routine postarlor/ 
anterior and lataral chest x-ray. 

Detarslne tha syBtatnatlc mathod 
for exainlnlng the roentianograph, 

Nanie atjeast six posltlona and/ 
or technlquas for avaluating the 
chest by radiograg&yi 

Interpret the slihoyfitte sign and 
describe the anaCondCilocatlon of 
chaat lesions utilizinlfthe 
silhouette elgn, 



13 Intefpretation of 

(8/5) Chest X-ray 

(1/1) 



(.5/1) 



-5/1) 



(.5/1) 



Name prifflary slnd pmi^ary signs (,i) 
of atelectasis, 



8/5 



0 



70 



Dofina the air bronchograiii. 



(.5) 
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3rd Ssfflsster (cont'd) 



Block Paragraph Objsctlve 



2h 



cont'd 



21 



2j 



2k 



21 



2n 



Idintify radiographic abnormalities 
associated with diseaies of the 
pleura! atrapleural space, and 



apnrai 



Present 
Houri- 



Nue criteria for differentiating /-(J) 
alveolar and Interstitial lung 
dissases by x-ray, . . / 

Intirpret rointgenopaphlc features 1 
to deteriini the nituri of the sol- 
itary lislon (benign or naliinint) 
of the lung. 

Idintify caiciflcation within the ,5 
lung. 



1 



toe the thru arbitrary compart- .4 
nents of the niediastinuni and 
structures found in each compattiQent, 

Describe the chest x-ray and lung .3 
scan signs of puliionary enboll. 

Delineate the lajor signs of chest .3 



Subject 



POMC 
lours 



0* 
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22a 



Describe the actions, effects and 
ilses of the antihistaiine drugs, 

EKplain the actions, effects and 
uses of drugg used to coibat 
infection, 



9/5 



Hlatanilnes gnd Anti- 
hlstanlnee 

Cheinotheraphy ^ 



2/1 

9/5 1 



3rd Sen (cont'd) 



I 



Present pqmc 
MlE!£l Objective ' jur^ yje^ flours 

^ ^ ^Isrgy k ' AUsrgy (Iiiiun. Mech) 4 

13b Deicrlbe the syndro^and treatoiint 3/1 Infectious Diseases 3/1 
of coryza, pharyngitis, bronchitis, 
and pneumnla, and nan the charac- 

teristlcg of bacterial and non- * 
bacterial pneuionia, 

u 

13d Discuss iononucleoais 1 InfectiouB Diseases 1 

(Mono) 

21a 'p PulDonary Diieases 28 Pulnionary Dlseasig '20/8 

(20/e) 

21a Describe special procedures appll- 1 
cable to atiologic diagnosis and 
esaantials of data base. 

21b Define lung function and normal 2/1 
pulraonary anatonyj and describe 
the physiology and pathophysiology 

of the respiratory system. 

ft 

0 ^ 
21c taa reasons for perfonnlng pulmDnary 3/1 

fynction testing and describe : 
/ routina tests of pulnionary function, 

name four mechanliois of hypOKeniiaj 

and how to detect and differenElate 
* theee iDechanisniSi 



2ld Name and define catpries of 3/2 
chronic obstructive lung disease 
and describe pathophysiology of 
' cystic fibroais, 



3rd Ssi (cont'd) 



Paragraph ObjBctlvg 



Presenc 



(cont'd) 



21f 



21i 



2ih 



211 



21 j 



Define respiratory failure and 1 
describe the physiologic effects, 
€tlolo|y, diapostlc teits, therapy, 
coipllcatlons and prognosis, 

Define throfflboeibollsD and infarc 1 
tion, and discrlbi pathophyalology 
and treatBent, 

Describe prliary alveolar hypovent- ,5 
iiatlon, alveolar proteinosis, ind= 
desquamative interstitial pneuionia. 

i' 

Describe diseases associated with ,5 
^ kyphoscoliosis and ankylosing . / 
apondylitls. I 



Describe Goodpasture's Syndroie and ,5/, 5 
Ideopathic pulnonary heniosiderosls, 

Describe lung abscess and diseases .5/, 5 
of pleura 



21k 



21i 



Describe the physical signs and 2/1 
spptoffls of aerobic and anaerobic 
bacterial pniuioniaB, and inycotic 
diseases of the lungs, 

List types of neoplasins, describe , I /J 
priinary tuiors of the lungs, and 
discuss the significance of the 
iolitary nodule. 



3rd Set (cont'd) 



Block Parairaph Objective 
21ii 



XXI 

(cont'd) 



ffi 



2 In 



2lo 



21p 



List clinical fiaturis of pulionary 
fibrosis and inscribe the radio- 
graphic picture and pulmonary 
function abnonalitlej associated 
with diffuie interstitial fibrosis 
and the Ham-Rich syndroM. 



Present 
Hours 

1/.5 



'Scrlbe Inhalation dlseaees of'tlia ■ 1/J 
lungs due to orpnic and Inorganic 
duBts, noxious gaseSi and' other 
agents , 

Describe apiHeBdologyj causative' 1/J 
organises, imunology and ttaat- 
jsent of tubarculosis, 

' \ ' / ■ 

Deecribe tha pathogenesis and 1/ 

clinical aspecta of sarcoidosis, 

Wegener^s granuloMtoslSj and 

eoeinophlllc granuloMi 

Describa coniioon respiratory prob- 2/1 
im in childhood j their evaluation 
and manageWnt 



Subject 



POMC 
Hours 



0^ 
1^ 



Pediatrics (Rispira- 2/1 
tory dlSi of childhood) 



/8 



7 



TOTAL HOIS! 



52/20 '/^ 



ERIC 



65 



APPENDIX C 



POMC Evaluation Plan 



ERIC 



PROBLEM ORIENTED MEDICAL CURRICULUM 



EVALUATION ?hAH • \, " ' 

(prepared by AFHM,/TT as a result of Jraluatlon con|lerence held 6-7 
'Mar, 75 at Sheppard AFB) / ^ 



BACKGROUND 



Under the ai^plces ^cjf the Advanced Research Projects k'gency (ARPA) 
and Air Training Connnansd^ (ATC) , a Problem^Oriente^ fjedical Curriculum 
supported by PLATO IV technology is , being Introduced in the School of 



ed instructional 



Health Care Sciences, PLATO c#n^§rq^ide iBidivldyaMz! 
material presented through a computer 'contralled pl^ma panel . The 
essential hotion behind the Problem Oriented Medlcal^urriculum (POMC) 



is that ' academic subject matcer should not be Introduce'd in isolated 
units, but rather in the" CQntext^^^^^lftiinical , diagnosis of actual patient 
problems. In this fashion, it is anticipated that problem solving skills 
will be developed to a greater degree^ that retention of inf orTtiat ton will 
bo enhanced, and that student motivation will be Increased. Details 
cuvering Introduction of this innovation Into the Physician's Assistant 
Course can be found in the POMC Implementation Plan, Vfl" Feb 75. 



PROBLEM 



rhe ^\o£il of tills evaluation n^l^n i^ to develop a set of procedures 

f 

tor ass es s i ng t h e o f t c c t i ve n es J o f t lie p ro pos e d POMC , 



RATIONA LE 

11 le initial step in tip pruceas is to define a .rationale which 
undci lies and supportH th| entire evalUcUion, Proper eval ua tiky^is 
u 1 5 V i tly 54 ty^^ c o mp 1 0 X ma 1 1 e r , f o r 1 1 1 e c o n c e p t o f " e f f e c 1 1 v e n e s s " is 
elearly mu I t i d i mens i o nn 1 . However, two sets uf questions seem^ to be 
generated almost Inmiod la tel y , The first ^iroup is primary and b^ically 
oporationai in nature; theae quesi . una =aju designad to create a rational 



\ 



tPy for or agalnsfe full flcaie impiemantatlon of the Innovation. 
Simply \«tated, this set of questions . asks i / ' 

■ . cam.lt be 'don(i7= : * V ^ 
is It inatructlonally effectivte? 
^ ■ * what does It cost (to lnatitut|j to bperate, to malataln)? ' ' 

. is it reliable? - ' ' . 

^' \ " ■ ' ■ < ... ■ 

A secondary, or reeearch-oriented group of queatlons la also of 
interest. These are directed at explaining the euccess or failure - 
of the Innovation in the particular eiivironment studied. This set 
consists of asking ^ ' 

hpw was it doTie? ^ " . , ^ 

, > why was^ it effectdve? . * 

how could It be Improved (In terms of, costs lowered or ef- ■ 
fectlvenesB increased)? ] ^ 

^rhese latter questions are not unimportant, but higher priority ijiust 
be given -to the first aet ^ce they were Judged to be mo rd' relevant to 
the purposes qf >the sponsoring agencies (ARPA and ATC) , To so^ extent 
1^ '^th^. seoondary questions will be a4dresaed in the final feport, espeoially 
^with regard to describing the process involved in setting up,, the POMCj 
but the major, emphasis of available relources is being placed on answering 
the first group of questions, " . . 

^ Another ls.^ue, which was faced in planning the . evaluation, concerr 
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the extent t^fcfe^^^ 'comparative measurement will Ve made, as distln- ' 
..^ gulshecLfr6m following purely descriptive procedures . Thife distinction 
' is analogous to the difference between nortns referenced and criterlofr-V yN 
^ referenced ^asuramerlt, In tH^ cbmparatlve approach, procedure A is- . - ; 
^ compared with pr8cedure B various /criteria of ^ effect^JAies ^ile 4 
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in the dsicrlptlve approach, effectiveneas la measured with respect to 
meeting fltan^ardfl. or apeelflcations In aome absolutB sense. Each approach 
has Itf own pecirtlar advantagas, but, where feasible, it was decided 
to use comparative measurei as much as possible in order to contrast the 
POMC approach with traditional instruction. 

Another conceptual coneid|j|ation important for this rationala deals 
with focusing on which aspects of the situation should be studied. Although 
not erfiaustive of the possibilities inherent In this situation, three major 
aspects were identified fpr intensive study--.students, authors, and PLATO 
IV itself. ' \ ' 

A final concern involvea deciding which apeciflc characteristics and 
varlablas need to be measured. Since student and author performance are 
crucial to judging ef f ectlvenesa , these naturally are emphaslEed. But it / 
was also viewed, as ^portant to measure attitudes and to sample opinions ^ 
from the participants. In addition, concern with author background 
char ^teris tics and their relation to performance was seen as both re- 
searchable and Important for future implementation. With respect to the 
PLATO device, its various cost factors and overall reliability must ba 
addressed. 

The foregoing considerations gave rise to Fig. 1 wHlch outlines the 
major atudles to be completed as^part of this evaluation, t^o further 
constraints which influenced the choice of particular studies were (a) 
the necessity to work within the guidelines of the' POMC Implementation 
Plan, and (b) the limited manpower reeourcfs of both the PLATO Devel- 
opment Branch and AFHRL. ' . 
SAWLING CONSIDERAT IONS , • 

For those studies involving comparative evaluation of students, a 
POMC test group (numbering 16) will be selected from personnel entering .. 
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June 1975* These students will be presented specific problem 
oriented material on PLATO IV, rather/ than receiving the "equivalent"- 
instructional material in the traditional classroom setting, These 
macerials^ will be distributed throughout the entire three sefeesters of- 
claaswor^ at the School of Health Care Scdyifcices (SHCS) * The rettainlng 
48 students frdfa the June 1975 class will serve as a control group and 
will not be eKposed to the problem oriented material related to 
Respiratory Disease on PLATO. IV. ' 

The POMC teat group will be randomly aelaeted from the PA students 
entering in June 1975. If the sample chosen, by this method does not 
differ significantly from the cdntrol group on SAT composijte scores^ the 
Bample will be accepted as representative. If there Is an appreciable 
difference between the control and test group on SAT composite scores ^ a 
new selection of groups will be necessary since it has been shown that 
there is a high correlation between high composite SAT and success in the 
PA course. If a second selection process is necessary ' the entire class 
will be rank ordered, on SAT scores. Ai^equal number of students will be 
randomly selected from the high and low strata. Sample size can be 
increased to 32 by following the same procedures for the next entering 
class.i thus increasing the power of this design. 
STUDENTS 

1* Academlc__Knowledge (Study 1 ) 

The purpose of th|la comparison is to determine whether there are 
differences bitween POMC sturfents and students in the traditional 
curriculum with respect to academic knowledge of reapiratory di^ase 
subject matter. *If the POMC succeeds in teaching this material at 
least as well as currently, then academic quality will have been 
maintained. 

. 8o ■ 



since slight differences in emphasii and content are found between 
'the POMC and the traditional curriculum, academic test items from 
currently used block tests could be unfair to. POMC students. Likewise, 
the POMC^geherlted^end-of-biock test item could be biased against the 
traditional students. To resolye this difficulty, a test covering 
respiratory disease will be constructed consisting of IteM that are 
similar to those found on the American College of Medicine/Board 
Examinfftion* These items will come from the Ed|jcatlonal Testing Service 
(ETS) item banks. In the event that ETS cannot supply theae items or 
morie items are necessary^ experts in respiratory disease from outside 
the SHCS (for eican^le, Univerrfity of Nebraaka EA s.taff) will be asked 
to contribute test items. These procedures wi^* insure content varidity 
since these test items will tap knowledge .that la copldered Important 
by independent ^perts in respiratory disease, . - 

, This initial Item pool will be further refined by testing a group 
of Physician's Assistants Just at the end of their second year of tralnin 
Item statistics will be computed on this sa^le,^ and Items for the final 
form of t^^test will be chosen in accordance with statistical criteria 
so that only reliable and discriminating Items of appropriate difficulty 
are included. In addition^ this procedure will provide a normative sarapl< 
against which scorae of the test and control groups ban be compared., 

The final form^ this test will be administered to'^test and control 
groups at th^ end of phase I .of the course, thus delayed retenrion will 
be measur^ for both groups. < . ' 
2.. . pWblam Solving Sklll-CStudy 2) \ . • 
The skill to be assessed Jn this comparison Is problem s^plving 
or diagnostic skill. By presenting several simulated patient encounters^ 
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we can test the student's ability to integrate his knowledge toward 

succeBsful dlagnoais and treatment of a patlant. These exercises will- 

essantially be paper and pencil *probJ.em descriptions > but the studant 

will be able to request Information, prescribe treatments* and note 

effects. In a limited since he will be able to ''Interact" with the 

simulation. Scoring criteria would Include correct use of available 

Information (lab teats, patient history^ physical examinations, special 

tests, etc), correct sequence of st^ps toward dlaghosls, accuracy of 

diagnosis, appropriate indication of treatment, time to completion. 

Two or three Simulated Patient Encounters will be presented to 

4 . 

poth groups at the end of the course. It Is planned that these 
eiraulatlona will be availably from ETS or drawn from slihulations which 
are already in existence. It Is felt that creation of simulations 
by the PLATO and PA staffs should be'avolded to eliminate bias, 
3. Student Attitude Questionnaire (Study 3) / 
^ The main objective of this study is to discover whether POMC 

studente have positive (it negative) attitudes toward the curriculum 
and/or the PLATO system^ To this end, a 26 Item version of the 
Brown questionnaire, used by Florida State University in a recent 
study, has been selected, ais instrument has demonstrated reliability 
with college students (r*,80), and includes many specific questions 
which will permit infarences out the varioup features of PLATO (e.g. 
p^ceived reliability' of terminals, depersonalization, etc). 

It is anticipated that the attitude questionnaire will be given 
in the first and twelfth month of- the course in order to detect 
differences as eKpetlence with the POMC and PLATO increases. Analyses 
of the relationship between attitude and perfornmnce on the final 
academic test and simulations, are planned. 
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4. Lesson and Block Teats (Study 4) 

Here the evaluation will be descriptive aAd formative in contrast 
to the previous study which was comparativB and summative. The goal is 
to show that POJIC students are meeting learning objective^ ^they 
progress through the course. Number of ley on repeats, number and 
duration of special, remedlations , eliminations or washbacks will be 
reported, 
^ AUTHORS 

1. Performance (St uc 

Author performance will be measured In' three ways I ta) av^erage 
quality of lessons pre duced (as judged by University of Illinois 
Personnel and student! (b) supervisory ratings of performance, and (c) 
peer ratings of perf or/nance. Performance dimensions to be rated will 
include progr^imJj*rftlllty (competence with Tutor language) , efficiency, 
in lesson ydTslgn, creative use; of InstructlonM techniques, novel 
applications of PLATO, etc. ■ • 

it- 

questionnaire instruments will be designed by AFHRL for administration 
as soon as possible, so that time comparisons can be made to show the 
effect, if any, of experience. 

2. Attitudes/Opinions (Study 6) ' 

Attitudes of authors will be measured primarily by a question- 
nalce yet to be developed. Two purposes are paramount here— '(a) to be 
able to describe specific strengths and weaknesses of the PLATO equipment, 
and (b) -to. investigate the relationship between attitude and performance. • 
Dimensions to be sampled Include: attitudes toward CAI In general, 
feelings about Individualized instruction and the problem-oriented 
approach, human factors aspects of terminal design, problems^ in lesson ' 
design, etc. 



Questionnaire data should be supplemented by intfrvlew data with 
interviaws to be conducted by ^hRL, Also open ended opportunities will 
be Includad in the questionnaire format. * 

Two measurements of attitudes and^^perforiMhce will be made at different 
times. Cross-lagged correlation analysis may yield some Insight into 
whether attitudes cause performance or vice versa* 
3, Characteristics (Study 7) 

An attempt will be mfde to relate author personal and background 
characterises to performance on the project, . To this end, data will be 
collected on such variables as level ind type of education, ages rank, 

status, Inteteats I ^etc , The aim 
here is to predict what kin,ds of people make successful authors for 
possible selection purposes* ' . = 

PLATO IV ^ J 

I . Cost (Study 8) . . 

Since the PLATO Development Bra^|| is a cosf center^ aggregate 
/^operating and development cost figures can be readily obtained; >knagement 
analysis personnel will be contacted for consultation purposes. Capital 
■Investment figures should be available from ARPA. A projection of 
/expansion coats under saveral scenarios should be a part of the final 

report. » . . ^ 

/. . . ■ 

t ' 2. Reliability (Study 9) - \ ' 

^ - - - - 

^ Logs will be maintained to show number of terminals in commission 

}|5ar day, percent of down-^time per terminal, cause of failure. Time phased 
analyses can be made to detect trends p 
SIM^_RY * \ 

Figure 2 gives an overyiew and summary of the plan with responsible 
agencies and deadltnes specified. 
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AN EVALUATION OP THE BDUCATIONAI, AND COST EPPECTIVENISS dP THE 

PLATO SYSTEM ' ' 

1. Introduction , ■ • 

■ " ■ ' ■ s 

The CERL proposal, "A Coordinated Evaluation bf-the Use of PLATO at 
Military Training Sites, ", dated 17 75, cites the basic question toward 
which research must be directed as: "Under what conditions can CBE make a 
effective contribution to the Instructional naeda of thi servlcea." ■ 



cost 



On the basis of the e^erience gained through current research effort into 
th* development of a new problem oriented curriculum at the physician assistant 
lavel, using the PLATO IV system, the School of Health Care Sciehces is In a 
uh^que position to assist In research of a subset of this question, specifi- 
cally, "can applications for the PLATO IV system and related Instructional pro- 
grams, be found within a military training environment which are Instructionally 
effective and cost effective." 

_Sh^pard capability to pursue this queatlon centers on the fact that 
a trained st^f is already In place; physical facilities, including a PLATO 
classroom wit^ necessary teminals, carrels, cownunieations equipment, power 
and supply Already exist; and the SHCS offers a wide varfety of courses designed 
to provide training in numerous specialities at all skill levels using a wide, 
range of instructional techniques and stratagies. Therefore, the SHCS offers 
an Ideal environment in which the continued, exploration of the effectiveness 
of CBE can be evaluated. 

II. Rationale 

As noted previously, the examination of both the educational and cost 
•effectiveness of the PLATO system In military training Is a question that must 
be addressed. The current research effort is not designed to meet this need for 
a ntober of reasons,, specifically, it primarily ■addresses the impact of a major 
curricular modification in a course that Is unique in military medical training. 
Additionally, the highly specialized nature of the Physician Assistant course 
and Its inherent time constraints prohibit the demonstration of either actual or 
potential time, or cost savings, 

A jDreliminary survey of courses conducted within the Departments of Medicine 
Dentistry, Nursing, Biomedical Sciences, and Health Care Administration revealed 
a number of existing courses which appear to be well suited to investigation of 
the cost and instructional effectiveness question cited earlier. 



Ill, Proposad Ragaarch Dasign 

The major issues of cost and ingtruetional ef faGtiveness can bast be 
addrassad by a series of individual* but wall Intagratad studias. The key 
to the success of any research in this area is the seleotion of a aoursa in 
which optimum potantial for oomt and instructional af factivanass is -perceived. 
Additionally^ it must be undarstood that the amphasis of any research under- 
takan at this time must focus on avaluation aspects rathar thah an aKtensiva 
courseware developmant effort, ' 



To insura idantif ication of the optimum course j a nmtdper of tentative 
* selection criteria ara suggested below. Criteria were selected to minimiza 
implementation impact, maximize data collection potentialj and utilise existin 
rasourcis* Proposed criteria iricludai 

a*" maximum class size . , 

b. trained personnel requirement (TPR/for year) 

c* course length ' * 

d* class entry schedule 

e* instructional methods (lecture* self-pacingj etc) ^ 
f , PLATO staff ^ckgrounds 

hardware resources ' ¥ 
h* homogenity ^and interest of traditional faculty - . " 

i. transferability to civilian coi^unity 

Preliminary application of these criteria to the courBes currently being 
taught at the SHCS resulted in ten potential candidate courses which ware de= 
serving of addi* 'nal evaluation and consideration. These courses ares 

Course Dept Length TPR Max Gp Size Entry Schedule 
Radiology 

Specialist MSDM . 14 wk 237 24 ' , 4 

IDT MSDM . 9 wk ^ 32+ 14 13 

Diet Therapy MSDB ' 51 wk 112 12 3 

MSDB II wk 373 , 32 3 

MSDM . 8 wk 228 16 3 




I ; 



Course 

Med Admin 

Spec 

Med Mat, 

Spec 

Med Rac, 
Mgmt 

Dant , BpBC 

Prev*» Dent* 
Spac 



Dept 



HSDH 



79 



Length tor 




6 wk 

6 wk 
Si wk 

9 wk ; 
7i wk 



232+ 

265' 
90 

255 
90 



Max Gp Bimm , ^ Enta^ si^^la 
20 ' 2 ^ 



2a 

12 



3 ^ 
9 

3 ^ 
6 



IV, Proposed. Studies ' \ . ^ ' * 

Studies which can be accqimodatfed under this proposal will provide CEHL 
and the sponsor valuable data with which to address many of the questions ^at 
forth in tha p^oposall Specific studies of interast^to MC, AFHMi, hifiK^i 

CEHL and SHCS can be designad to address tha following topics i ' 

a* CBE lasson af fectivenaas versus lectures ■ 

b. CBE lasson eff ectivenass versiis other salf-paead madia^ 

c, effectivenass of alternate instructional stratagias in CBE lassons, 
d* lesson optimizatiori in relation to time savings 

e* cost effectivenegs determination ' 
f , ' attitudinal data * . - 

(1) .students 

(2) CBE authors 

(3) traditional instructors 



In order to avoid the evaluation problems which impact the ori-going research' 
effort, j every effort will be made to insure that comparable cost and effectiveness 
data can be collected from both the proposed PLATO application and the ^ traditional 
(controi) course, ' ■ . ■ ^ " 
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Resources Haquirad 
Persdnilel ^ ^ currant authorisations Xess Piiysician " 
Equipinent -r mil required equipment ie eurrenfely in placa 
Time ^ 1 yaa^ * - v. " . 

Cost Approximate ly $ 9 OK (Comiimic^t^ns s^; Computer' Supporf) 
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■ ■ . ' ' INTHCb^UTION;"; ,'' ' \l'^ -'': 7t'\'' ', 

^ ^ This chatpter cnltiques ShappardVr pf obl6ra-orientedi4ed tr' 
Vcunrlcurum'GOTrseware/ TWe pro^leln wi^h rt'^iewing^v ' 
grouj) of lesaons>tbgetheF is that it,''ff^lffl,cult to tfrik-in 
anything but avera^ges and Eenerallt^^ ^ Without ^eadr7ibin'E^^ 
eioh lesson separately, it's haf'di^tfa eonVey th«^^arl/^lon^ . in 
; apprpiGh and' quality in this potpourri oS^ both ; thf /ore,ajti^ 
^ and the eommonplaoe. ^The ^heppard prcrject was pam 
staffed by some of the most intelligent, cpnsci'ei^ , 
authors atf any ARPA/pLATO- site * ^ These authors prod . / 

inventi^^e lessons, adapting well tp ;the uni^que fe^ 
and necessities inyolved in using eomputir-based approaetk 
to' education , Other l^tKprs clung to ol;^^^^^ transfer* 
rii%&^ lectur^lfkei technique to PLATOV;/ fioi?^^ had no 

. ^aclfi^ experiendg but modeled 'their aft^r the sort 

of ig^,tructlon they: themselves bad b^^it: expc^ed to as stu-- ^ 
dents* Some authors* weloojr|ed t,he. chaUenge/ while others / ^ 
resisted the change Attitudes and cofnmenta ranged from, "I.' 
was taught by lectures and struggled through s© why' should ' 
other students feel entitled to sdmethlng to, "Les- 

son writing is a tremendously 'creative, personal process, , * 



it?s like giving birth!" 



What this chapter describes, then, th^ middle ground, 
the lev^l where most ^authors operated^, wJfi?re most lessons 
"worked". The point, after all, #as notJto "be creatlvei^ b»ft 
to produce apprdxlmately 150 student coi^tact hoars of dompu- 
vter ized instruction . v 





■ ^ ' . . ' . ^ ^PROBLEM ORIENTA^ON ^ :\":^v = = ^ 

, was disadssed ia "Cr itical An^lysii theX Inl^ . 

^ Phase of thi Sheppard AFB CBE Project*^. The Problem Orii/nted: 
Medical CurriQalum" , Missaft, 1978, a single , officialS^efi>- 

' nit ion of problem-orientation was never: really made, Conse- 
iquently , thei^'e were as many ORinlbns \of\ the meaning of pro- 
blem-orientation .as^there w^Xj authors. ^ Sheppard courseware 
reflected this divergence of opinion. Shrpp^ard lessons were 
usually one of three typesr / 



1, content-oriented 
^. ' ^ . 2,; scenario-oriented 

^ - 3. case-oriented. 



All four types dispense inf ormatipn ^ but wl€h differing \ 
emphases or foeal pplnts. In the flr^kt case ^ information is 
presented in a straightforward., almost lecture^like fashion. 
Ttre emphasis is^ori lesson content only. These lesson aut,hors/ 
felt 'the curriculum. i^Quld (and^should) be problem-priented as* 
a result of other kinds of lessans.^. These authors did not 
consider their con^tent-or ienteB lessons part of the problem- 
oriented- approach , ' ' . ■ ^ ^ ^ ^ 

Authors^ using a "scenario" approach often in^rQduced the 
lesson wltfh .a clinical aepictlon or situation. For example 
one desso'ii Opens with the fol lowing scene- - ^ / 

• - . ■ ' ^ ■ ■ ■ ; ■■ ' -s/i; 

^ ^ ' ^"you^re walking on the surgical ward and as you ■ " j 
pass a room you hemr a" yell and a noise like a - /^'^'^ 
. person falling on the f f oor . You rush In and; ^ 

" ■ - ^ i'- 

there on the flo^r is.a patient (45-50 year old^ ' 
male)- not brea^thlng, no heartbeat ^ no blood' > :^ 
: pressure present. You 'immediately beftin CPR / V/ 



(cardlopulmonary,r«es.uaoll«tlon) and call tor 
help, but in apltt .)r an the .6f£ort to-jie- 
suscitate thm pa,tl^«t ^« is prbnounced, dead. 
'As you get up"y^ -^ie* a bedpan on^' the bed , 
WHAT HAPPENED? . ' ' > 



■Throughout ■ the remaindffr of the lesson, thf author used the'^ 
openinp setting as a springbdard' to intfrod^ce"reievani ^for- 
' mation, frequently refering bAck to the c]/inlcal seene^^'l How- 
e\ner, the student is not expecyd to diagnose or "solAre" the , 
case . e I 

Finally,, authors sometimes us^d a case-orlen-ted approaGh. 
The third trimester was to be almbst totally comprl^jd of 
.simulated patient encounters in whtch the- student was expected 
to diagnose and treat the conditl.ori\. In addition, some les- ' 
sons centered on less advanced sl'mulations In which, rather ■ 
than diagnosing and ■^reatlng the cask the student had only 
|0 assess the fCase in some other way. a For example, in one 
llsson the student Is presetited with a\i imaginary dialogue 
between a patient and doctor. The stud^ent's bask Is to 
determine whether the Information the pitient gives does, or 
does not contribute to making a diagrtoslL If the'patient 
information is, iqdeed Important the lessen branches to a 
brief discussion of the Implications and ■ iWortanoe- of that' 
piece of Information. In a lesson on , how take a complete 
patient hist'ory, the student again- sees a pitient/doctor dia- 
'logue,/and must decide whether' the patient lijformation is 
subjective or objective. 



•An aside for those who are curious about the \f ate of our . 
patient. Straining to use the bedpan had caus%d\massive pul- 
monary emboli--ia common concei^ witjn pdstoperatiye patients. 



LESSON DEVELOPMENT 



Studenp Population 

. ; / Stjfeppard Physician Assistant students are 
group^^ of highly motivated, hardworking peo'pia^. 



?mogeneou9^ 
A^T^^stu- 

dents are nonoommiesioned officers with 3*6 yea^s 6f,prW 
vlous medical experience. They must also have completedKa/ 
Medical Service Fundamehtals course and Jbraln^ng at a tech* 
nical school* Thus, PA students are older than niost miliary 
students, and serious about med icine as a career both in/ and ^ 
out of <the military. For this reasW, they are generaQ^y 
self-inotivated and, as one of fleer stated , "//.study as Ipng 
and hard as MDs." ^ . ^ 

Cognitive Le v^s ^ . ' v 

The SpecTalty ^Training Standards^ (ST? 917x0 for AFSCi, 
91730/70, 23 February 1972) for the Physician Assistant . ' 
course prescribe student objectives at 3 different levelst 



1. task perfQrmance 

2. :^ task know^ledge 

subject knowledge 



The following section is taken directly- frorn the STS 
"qualitative requirements*' for the Physician Assistant 
Course. It lists the general cognitive levels and perform- 
ance skills spec^ified In the STS. The remainder of the STS 
.expands on all areas of the course, enumerating which of 
these skills apply to what areas. Since the vast majority 
of th.e SKeppard/PLATO material was aimed at content rather 
than performance^ only the objectives f©r task and subject 
knowledge have teen Inc luded h^re * 
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task Knowledge 

a. Can name parts, tools, and simple facts abaut the task. 

, {NOMENCLATURE) , , * ' . > . 

b. Can determine step by step procedures for- doing the taslt. 
( HROCEDURES) : \ " 

c. Can explain why and when the task must be done and why 
- ;each step is needed. (OPERATING PRINCIPLES) 

d. Can predict, identify, and resolve problems about the 
task. .(CQMPLETE THEORY) ? ' . ^ ' 

Subject Knowledge . . . . . 

A. Can identify basic facts and terms about the subject k 
CFACTS)^ _ ^ . i 

B. Can explain relationship of basic facts and state General 
principles about the subject. (PRINCIPLES) 

C. Can analyze facts and principles and draw conclusions' 
about the subject (ANALYSIS) 

U. Can evaluate conditions, and make proper decisions about 
^ the subject. (EVALUATION) 

Sheppard lessons generally emphasized objeatlves airned ^ 
at nomenclature, facts, principles, and proeedurea* The 
analytic, evaluative, and problem solving levels, were mainly 
concentrated in the simulated pa tient encounters. Though a 
small numb^er of lessons required the student to either apply 
principles to new; si tua tions or synthesize information^ and 
form a conclusion, most lessons expected only that the st^uder 
be able to list or restate Information, name or Identify 
structural components, list causes/effects, etc. For example 

--^State 4 factors ^that affect the , diffusion of gases. 
— 'Name 3 methods of sputum collection. 
--List the agents/factors responsible for upper . 
airway diseases/disorders. 
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--I^intlfy and name th^ bones that make sUp the ^ 
, \ . thoraalc c^ge. ^ V ' 

'-^^Riine the 4 Jung volumes and match each name to it's 
' ' raspeotive volume. \ . 

are objectives taken f\ror^ POMC lessons which typify the sorts 
of tasks and levels understanding required 6f students, 
Certainlw some aspects of clinical saience 'Icivolve rote 
memorization, and. of ten all toat needs to be tested Is ^ 
whether 1 the stud#n^t has actually committed them %o memoryV 
However I the problem .solving skills alluded to In both the 
STS and course definitions, of the POMC were only irftermlt-' 
tently reflected In actual behavioral objectives or stressed 
in the lessons themselves . Ont possible cause waS that objec- 
tives were frequently written after the lesson ItselT was 
written* Often tf it was pointed out that the objectives and 
lesson content were at varlanoe/ the author would rewrite the 
objectives. Many times authors seeme,d ftot to have a' clear 
idea either of what they wanted the ;Student to be abte to do 
by the end of the, lessoti or of what actual \on-the- Job skills 
the student needed. This may have stemmed In part from the 
fact that few authors did any sort of task /analysis before 
wrltm§-,^B4ther the objectives or the lesson Itself. This was 
particularly true of authors who lacked tf aching eKperlence. 

Developmer}t Procedures - 

All authors had subject-matter expertise in some area and 
wrote lejssons concentrated in their field. Some authors, 
however, also ended up writing lessons In areas they knew- 
little or nothing about. This took up valuable time while 
they had to research lesson material.. Also, as was pointed 
out In Sheppard's phase one report (Stelnkerchner , Delgnan, 
Waters & D%Leo), lack of subject matter expertise hampers a 
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programmer's ability to make ".intelligent Gholces" about dis- 

play ,V^f e^edback , etc. ' ■"' : 

All curriculum materials were on PLATO. There was no 
m'edia mtxturei When MTC reviewers pointed, out that some of' ' 
the material was not well- suited for computerization and • 
could- be more economically and effectively presented using 
■another medirum, authors insisted that all material had to be 
on line. For e^xample, the laboratory testing procedures les- 
■sons were presented almost in the form of an on-line handout. 
Laboratory procedures were detailed in a step-by-step fash- 
ion. Then the student was -often asked something like, "What 
.is the first step in doing a gram' stain" o)r some technical or 
procedural point. Some parts of these lessons did, however, 
include the underlying reasons for using various teehTiiques, 
chemicals, etc. Thus the lessons" were admixture of both • 
theory and practice, with rationale iriterlaoing pages (some- 
times as-'man-y , as' 3 full frames at. a- time) of step-by-step 
procedure. When asked whether as a Physician Assistant the 
student, would ever have to perform 'these tests, the author 
admitted s/he would not. When the MTC reviewer pointy out 
that it might then be more ef fee five to print procedures or) a 
hard-copy hand out (since they were so textual) the author 
said all material had to be on-line regardless of whether it 
belonged there. The question of whether some material could 
really be taught best on-line never seemed. 'tg b' part of the 
lesson '.development process, " , 

As was briefly discussed the- previous section , the ' 
chronologi-cal order in which authors wrote their "lessons var- 
ied tremendously. Toward the beginning of the POMC projecte' 
an, MTC member provided the Sheppard staff with" a copy of. 
Chanute's lesson development and^ documentation pl^an for - 
possible adaptation, to the Sheppard project. The'Sheppard 
staff initially a|reed that such an "official" policy would 



V be a good idea, ahd decided institute a var^iation 

Chariute^s plafi. After aHhilei however; it becaTne dbvious ^ ^ 
tftat no' s^andirdlzed development prooM were aetuafjly " 

, being followed. V Objectives were not beihg written, off^llncv 

/ prpplannlng was sporadic ,/ etc * After this wa^s pbihted out , a 
perioli of about two, months followed in whlA authors cori'Sci- 
entiously wrote behavi^al objectives before beginning pro- 
gramming their lessons* For .some author^ these obJeQ^tlves 
represenWd the only preplanning they ^id. Other authors 
wrote outlines or portions of lessons before ooding* FSnally, 
authors fell into the habit of writing objectives and.,cri* 
^ terlon tests after the lesson was coded , , Jn this way objec- 

tives ware .tailored to fit the complete^ lesson, Jhe/lack of 
'objectives or preplanning in at leist on# In^tanG^ produced a - 
lengthy, rambling lesson which-the author "wrotfe*' directly 
on-line- Since this was his first attempt at lesson wri ting ,/ 
he had not yet developed the repertoire of design skills nec- 
e^ary to extemporaneously write a lesson^ However, :he 
regarded lesson writing as a creative process tha^^ was only 
stifled by flowcharting, outlining, etc. As a result his 
first lesson took nea#ly a year to write and polish, and 
seemed at many points to be disconnected , with. sections which 
followed consecutively but did not build on each other. The 

, author^s belief that lesson writing^ Is a creative process is 
a sound one, and experienced authors are certainly afcle to 
write a fair amount of material on-line, Howeveri just as 
one would not expect a jazz artist to ad lib a piece of music 
without having a solid musical foundation discipline anfl 
knowledge, one could not reason§bry eypeqt that a "novice" 
author could ad lib a lesson without a solid instructional 
foundation of design and media experience* 
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• . -LESSO>J-. DESIGN ^ ' • , ■ 

' -0ul to the -range- of knowledge .arrd 'skill requlf-ed, ©f a 
physician^ .assistajiti, Sheppai^d POMc' purr^ulurn' oove^^ ^:wlde 
gamut of ■subjects pivoting m^alnly' around anatomy , baajberio-^ 
iogy^ chen^ttrjr and ^h^S"ibiegy, .all spj^ififf the reap tM-' 

.|ory sys4;ern. the fin ished leaaons fall into 3- general citW 
gorl*s^tho|e 'whosft purpose is to preSWnt lnform¥tilon,;,teaeh^ 

-procedures or present cliniQal Bimu^ationa, In additloh,. 
lessons can be further G&talogued^ in, •terms of the approaches 
authors^ used in fulfilling- the prepedin'g purposes. Sheppard 
lessons' employed main instructional formats,,: 



a, lectune/questipn^ 



b. drills ' 

c. tutorials 

d. simulations. 



Lecfeure/Question ■ , - i _ 

The lecture format is the most common l.y used' apprQach in 

■'Which content is stressed ovej- tasks or problem-solvingt The 
lesson usually unfolds in- a . 1 ine%r fashion , nith little ^. 
diverg^n^e from the .maln^ patfc^. The student Is il 1 owed'=^o .|q ■ 
forward and backward, pnd sometimes is 'able to -branch off 
into a. special help section. Cesson material is typically' 
dispensed in a lecture/question order' in Hhloh a number of 
frames ("pages") of text are first presented followed by a 
question or questions covering what the student has just 

-read. , / . ■ , , ' 

Drills , . ■ 

A few lessons use a: dHlfl approach in which the main 
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emphasis Is on givinrt the student practice at using termino- 
logVj interpreting graphs, identifying various structural 
components ( i .e, '?touah the sternum ^ in this^^drawing" ) , label- 
ing dsUa (is this Information, sub Jectlve or objec^tiye? ) , etc, 
Some'lessons use a true, drill appro.ach (in which the student 
answer^ a number of .items per concept), while other lessons 
use a modified^ drill approach ( in which the student answers 
only t item^per concspt). In at least one qase J namely the 
Anatomical Reference Terminology lesson, the material was 
either, taught or gleaned in some other way, and the Ipsscrn 
was basically a review. In most other cases,, howeve the 
subject matter is bri'efly presented at the beginning of the 
lesson, s(T"ter which the student is drilled on-what s/he. has 
just readi For example, in the lesson ^'^Anatomy of the Tho- ^ 
tacic Cage" the lesson first loofites and labels various parts 
^ of the thoracic cage. Then frrfe ^student is given a number of 
dnawings of the thoracic cage^ on wl^ch s/he must point out^ 
which struptures^ are which • - 

Tutor lajPs .... 

Another approach is the tutorial in which the student is 
"walked" through some procedure step by step. this .apprdach 
is sometimes used as a form of help after the student has • 
already answered a question incorrectly. It's also used in 
o.ther ca^es such as the lesson on spirometry in which the 
student is lerf through a detailed explanation of lung volumes 
and capacities as demonstrated on a simulated spiT^ograph. In 
other lessons, jthe student - is guided through' proper proce- 
dures for taking a complete patient history, doing speqi^fic 
1 abora tory r tests , ^tc.' 
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The m0st^obyious use o,f simulations in the Sheppard cur- 
riculum is the simul-atPd _patient encounters in which the stu- 
dent must collect a ^patient history, gather physical examina- 
tion and laboratofy data,, then diagnose and treat the con- 
dition. No new subject matter is presented, the emphasis 
being only on having the 'student "solve" the case. Since the 
pro'ject ended before tKe third trimester for which the clini-' 
cal simulations were scheduled, few students ^ever saw the 
simulations and the lessons were no t in ..po^l ished form at the 
end of the project. 

A few other Sheppard lessons, particularly the lessons on 
f^as :iaws / iricorporated a different sort of simulation , Some 
of these lessons tried to demonstrate various experiments to 
illustrate cause and effect relationships. For example, one 
lesson simulates the effect^ on a manom&ter of increases and 
decreases in air pressure. 
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INSTRUCTIONAL APPROACH-ES. 



Tr.e follcwlni sections discuss various instructional 
/ 

approa^ches enployed by Sheppard authors. Lessons are 
described ^in terms of the three main areas of test ing , inter- 
act ion ^ an^l Bt vie , ' ' ;^ - 
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All the, PCMC, lessons have end-of«lessori tests* There is 
^generall^ one item per concept or stLjdent objective. Test 
items are almost alwiys mOltiple choice^ and, like the^ prac- 
tioe items in the lessons themselves, mainly test recall. 
Because of the. small number^ of items, teat reliability (and' 
hence validity) is often low, it is our understanding that^ 
these tests were intended -to be norm-referenced rather than 
-criteriGn ref erence^ in' nature , Our analyses of the item 
respon-se data preserWil. on the PLATO system hav.e verified ^ 
that. Tany oP the items have difficulty " and. discriminat-ion 
Ind ices which fall GUtslde the ranges usually recommended for 
^ norn^re ferenced tests. Such items detract from the useful- 
ness cf th^^ t'ests and ^should probably have^been modified or 
eliminated in the test development process. Because of /the I 
extremely small sample C n^ 1 6 ) who were exposed to the PLATi^'^ 
= ins'4riJcti'Dn , however, the tests were neve;^ empiriciriy vali- 
dated, fleno.e, the only data that, were ever tfollec t|d/f or 
these tests were F.enerated in actual operational us the 
lessons. Given the hurried and limited nature of the test 
development, it i.% not sur pr ising that so many of the 4tems ' 
appfrar to be of 1 i tt 1 e va 1 ue.* 

Host authors did not believe that there was any need or 
place for mastery learning in/ the POMC. While, some authors 
thour^^ht the criterion test should be discriminating, others* 
thour?ht' %hat PLATO lessons could only measure short-term 
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retention, so there was no need or reason to measure' mastery • 
This attitude may well have been a contributing factor to the 
1:1 ratio of questions to objectives found in/most of the't''.. 
c r 1 1 e r 1 o n t e s t s . ' ' . / , 1 ,^iC"'^''^?A, 

^ / ' ^:^> 

Interaction , - / ■■ 

" Over the course of the Sheppard P0MC pro J^c t , /thr^-amount 
of student interaction (in the form of questions)' ^increased 
markedly. Even lecture-type lessons became mrfre/' inter spersed 
wi.th questions. However^/the majority oi querflons were 
restricted to virtual verbatim, copy-fram^ ^pa questions in 
which the student had dnly to reiterate sor^e bit gf termino- 
logy, list inf ormat ion/ or h^peat data usuad ly yppesen ted from 
one to three frames ^/before the questioV it self * rin one ^ 
lesson, ^ for example, /this definition is /^iv#i\^'for upper air?-- 
way 'd iseases : ^ / - l-Xi . . i ^ / " 

"UPPER AIRWAY DISBASEg S^'?' ^ 



is the collective term given, to Wl^^^^^ .and 
disorders resulting from s^ruetuf-i 
or dysfunctions of the,:. 



1 • nose^ / ^^ 

2 . parana^^al sirruses 

3 . pharynx ^ . 

^1 * larynx ^ 



*A frame is any screen display only in^iatfd by student" 

input . . ^ , _ , ^ . ^ V ^ 

-'^"^ . M ■ ' <i;'./^ 



■■ ^ ■ ' ■ 

I " ' 

Two frames later, the student is askedi' > j 

■ ^ ■ ^ . ^ . • ' ■ ..J 

"Upper Airw.ay Diseases Have been defined 
\^"Bs disorders or dysfunctions of thei 

(Please list the terms one-at-a-tlme .-^ " , 
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^ In another lessarn on pulmonary emboli, one frame states: 

^ V \ ■ / - > - ' 

^/ "In -experiments , 75% of the pulmonary, vascular bed ' 
may be gradually obstructed before pulmonary hyper- 
tension develops," / ■ . 

On the page Immediately following, the student is asked: 

'^'What % of vascular bed may be obstructed before 
hypertension occurs?" ^ 

^ - Though. most lesson questions (and criterion test items) 
are of a verbatim recall type, a small number of the intro- 
ductory tfasic science lessons present the student with chal- 
lenging word"prpblem type questions. In % lesson on^gas laws 
for instance, the student must apply Boyle's Law to solve at 
least 4 randomly chosen word problems of the following sort: 
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"An 11 liter ^oontainer of gas is initially at a pre- 
sSure of 14 atm. Wh.at will the^g^easure be If the'vol- 
ume is changed to 19 liters at constant t^^erature?" 

s Fee dback and help > In the majority (of lessons, feed- 
.fe^ack^for incorrect student responses consists of a^^gjheral 
hint* If fihe student misses the questioi^'^a second; time ^yhe^ 
is. then told t^e correct answer and told to type in the cot- 
reet rejsponse. The general helf) may- be one" of a number of? 
sorts. One type is, the form cue in which, rather . than giving 
a cognitive clue* the hint consists of the starting letter, 
number of words in the response, eto^ /For ' example , (n one 
lesson , the student must answer the foMowing quest ion ^ 



"If a nasal smear shows a large number of neutro- 
phil's and eosinophils y.our patient most probably 
has an with a superimposed 

If the student gives an incorrect response for the first 
blank, the feedback s/he gets isr 



"Two words" 



If the student an swers Incorrectly again ^ the next hint is: 



"Initials ♦a-r ' " 



Finally, after the 3 r d try the student is told: 



"Ans^^^er - allergie rhinitis" 



More often the feedback gives a content cue as in %pe 
following example* The student is given the question* 



' ■'. ■■0'^'^^'^^^°^ state, of matter are the moleoules ■ -i's^ 

'^,Mrn(S)'ving- MOST rapidly?" ^, 

If, the^ atudent 'aliswers incorrectly, the computer ' s response 
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■ • "Molecular motion in solids and Mquids Is limited 
.by the close contact of nelghhoring molecules," ^ ^ 
^ ^ -.. ■ \ • 

A small number of lessons include questions with no help 
on feedback whatsoever. An incorrect resFLonse gene\all^ ell- 
cits oply a noncommittal "Try again", " Pr,ess -BACK-^ f or hel p" 
or a simple ^rio". The questi*rr mukt be answered before the 
student is allowed to continue, but s/he is left more or less 
to her/his own devices in divining the answer. While this is 
not the norm, it happens with ehough frequency to deserve ' 
c ommen t . ^ ■ <^ . ^ ■ 

While many lessons offer- the stuSenfr "help" at a ques- 
tion, the hBlp usplly consi^sts of having the student presf r 
key which sends hir/him back to reread some text s/he has /'' 
already been through^. This^ seems to be primarily a fuaCMon 
of what cognitive level th/que^t^on is aimed at. , ^^tions 
aimed strictly at a recall^^aevel require little or^^o exka- 
nation (these are also the sorts pf questions ,wfi'ich have form 
cues for feedback). Questions that delve l|i€o higher cogni- 
tive levels;,; however, in which thi studer^rlnust apply rather' 
than recite information, necessitate mp/e'sophisticatea feed- 
back and hel^. For .instanGe , df the student has difhculty 
making a diagnosis In the clinical siplatlons, s/he is 
routed to a socratic sort of tutorial in which the Student is 
led through the pertinent history and physical examination 
data to a diagnosis. ! 
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Style ^ ' I \ / : . / ' ■ Y 

Fbrmat . POMC lessons f^enerally utilise' one of three sim- 
ilar formats. The basic structure is' a linear presentat^n in 
which al] students tak'e the same straight-lined path through 
the lesson . These ressons sometimes' include a "roadmap" or 
"index" which map^ .out the route ^the student will tak^e, but , 
neither allows the student to choose , her/his own pat^, 
second structure, a variation on, the linear format, is a 
design* in which * the student pin^-pongs back and f orth between 
sections of the lesion and a , choice-index from whiich the stu- 
dent chooses the ^section s/he , wants to study, hut must take 
all sections in the order in wh^ich they appear in the index. 
If s/he deviates from t hat it inerary , s/he's told, "Since 
this is the first time you've been through this lesson, and 
since you have not corripleted section x, ^ou Siay not yet ro to 
section y." An advantage to this approach is that the student 

9 

may, however, choose to review any parts of the lesson s/he 
has already seen, A third approach' is ^a completely autonomous 
situation in which the student sees an index from which s/he' 
can choose any section in the lesson in any order s/he 
wij^hes. 

' Graphics . The RrBphics in POMC lessons also generally 
fall into thre^e main categories. The first type is often an; 
animation (as opposed to a static illustration) which either 
demonstrates or depicts whatever concept is being discussed* 
One lesson, for instance, has an animation of a- manometer 
which simulates the effect of changes in pressure. In another 
lesson the author plots various spirometrical ly ■ determined 
respiratory pressures on a graph depicting the relative 
volume of each Dres'sure, Graphics of this sort enhance or ., in 
some way broaden the student's understanding of a situation 
or concept either b^ simulating it or depicting it from a / 
different Derspedtivtfl ; - 
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The second ca£e-|ory cohsists of stat4c'^"picture^" which 
main'3,y serve to ipu^trate the text. While some of these 
.drawings are interesting even en-tertad nlaa.,' they don ' t ■ ' 
;.,g-enerally .add to, the student's understanding or bank of know- 
ledge ■(except in! the sense that they "show" the ^^jfl4^hat ' 
, a, thing looks' like). An example is a laboratory procedure 
lesson which includes a drawing of|a laborafcory slide with 
drop o'r liquid. While the . drawing is of some interest it 
probably adds very Little to what the student already knows.. 
This is not' necessarily a oritlciSn. A textual lesson with 
drawings is more interesting than one without, just, as a b^ok 
With pictures is more interesting than one without. Hoi^'ver, 
removing the gr^ics would probably do little harffl in terms 
of the student's grasp of the subject. 

Another category is the medical illustration/ Sheppard 
h,ad an extremely talented medical artist on staff who created 
beautiful anatomical drawings. These drawin'gs made an' impor- 
tant contribution to the quality of a number of anatomy les- 
sons .' ^ . 

The final type of POMC graphic is the ."cartoon" type 
whiph 9an he^ best characterized as a non-realistic- or imagin- 
ary illustration which students generally regard as "cute". 
A'n example is an imaginary flower-l'lke drawing named Al 
Allergen' in an allergy -lesson which waves a petal' "hello" at 
the -student. Another instance from the- same allergy series is 
a set of three figures used in a game in which the student 
must tell whether certain foods produce allergies or not. One 
I'igure is a running man, another is a -sleeping ' man , and the 
third is a smiley-faced sun. While these sorts of drawings 
can make a frame interesting, if carried to an extreme, they 
can also annoy the student." POMC students commonly referred 
to one series of such lessons as "Sesame Street","and felt 
they were^a bit Juvenile. 
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